151-200 of 1319 results (7ms)
2018-02-21 §
09:18 <chicocvenancio> killed io intensive tool job in bastion [tools]
03:32 <zhuyifei1999_> removed /data/project/.elasticsearch.ini, owned by root and mode 644, leaks the creds of /data/project/strephit/.elasticsearch.ini Might need to cycle it as well... [tools]
2018-02-20 §
12:42 <arturo> upgrading tools-flannel-etcd-01 [tools]
12:42 <arturo> upgrading tools-k8s-etcd-01 [tools]
2018-02-19 §
19:13 <arturo> upgrade all packages of tools-services-01 [tools]
19:02 <arturo> move tools-services-01 from puppet3 to puppet4 (manual package upgrade). No issues detected. [tools]
18:23 <arturo> upgrade packages of tools-cron-01 from all channels (trusty-wikimedia, trusty-updates and trusty-tools) [tools]
12:54 <arturo> puppet run with clush to ensure puppet is back to normal after being broken due to duplicated python3 declaration [tools]
2018-02-16 §
18:21 <arturo> upgrading tools-proxy-01 and tools-paws-master-01, same as others [tools]
17:36 <arturo> upgrading oldstable, jessie-backports, jessie-wikimedia packages in tools-k8s-master-01 (excluding linux*, libpam*, nslcd) [tools]
13:00 <arturo> upgrades in tools-exec-14[01-10].eqiad.wmflabs were fine [tools]
12:42 <arturo> aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'DEBIAN_FRONTEND=noninteractive sudo apt-upgrade -u upgrade trusty-updates -y' [tools]
11:58 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-wikimedia -y [tools]
11:57 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-backports -y [tools]
11:53 <arturo> (10 exec canary nodes) aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'sudo apt-upgrade -u upgrade trusty-wikimedia -y' [tools]
11:41 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade oldstable -y [tools]
2018-02-15 §
13:54 <arturo> cleanup ferm (deinstall) in tools-services-01 for T187435 [tools]
13:28 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-tools [tools]
13:16 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-updates -y [tools]
13:13 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-wikimedia [tools]
13:06 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-tools [tools]
12:57 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-updates [tools]
12:51 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-wikimedia [tools]
2018-02-14 §
13:09 <arturo> the reboot was OK, the server seems working and kubectl sees all the pods running in the deployment (T187315) [tools]
13:04 <arturo> reboot tools-paws-master-01 for T187315 [tools]
2018-02-11 §
01:28 <zhuyifei1999_> `# find /home/ -maxdepth 1 -perm -o+w \! -uid 0 -exec chmod -v o-w {} \;` Affected: only /home/tr8dr, mode 0777 -> 0775 [tools]
01:21 <zhuyifei1999_> `# find /data/project/ -maxdepth 1 -perm -o+w \! -uid 0 -exec chmod -v o-w {} \;` Affected tools: wikisource-tweets, gsociftttdev, dow, ifttt-testing, elobot. All mode 2777 -> 2775 [tools]
2018-02-09 §
10:35 <arturo> deploy https://gerrit.wikimedia.org/r/#/c/409226/ T179343 T182562 T186846 [tools]
06:15 <bd808> Killed orphan processes owned by iabot, dupdet, and wsexport scattered across the webgrid nodes [tools]
05:07 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1426 [tools]
05:06 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1411 [tools]
05:05 <bd808> Killed 1 orphan php-fcgi process from jembot that were running on tools-webgrid-lighttpd-1409 [tools]
05:02 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1421 and pegging the cpu there [tools]
04:56 <bd808> Rescheduled 30 of the 60 tools running on tools-webgrid-lighttpd-1421 (T186830) [tools]
04:39 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1417 and pegging the cpu there [tools]
2018-02-08 §
18:38 <arturo> aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1002.tools.eqiad.wmflabs [tools]
18:35 <arturo> aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade jessie-wikimedia -v [tools]
18:33 <arturo> aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade oldstable -v [tools]
18:28 <arturo> cordon & drain tools-worker-1002.tools.eqiad.wmflabs [tools]
18:10 <arturo> uncordon tools-paws-worker-1019. Package upgrades were OK. [tools]
18:08 <arturo> aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stable -v [tools]
18:06 <arturo> aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stretch-wikimedia -v [tools]
18:02 <arturo> cordon tools-paws-worker-1019 to do some package upgrades [tools]
17:29 <arturo> repool tools-exec-1401.tools.eqiad.wmflabs. Package upgrades were OK. [tools]
17:20 <arturo> aborrero@tools-exec-1401:~$ sudo apt-upgrade upgrade trusty-updates -vy [tools]
17:15 <arturo> aborrero@tools-exec-1401:~$ sudo apt-upgrade upgrade trusty-wikimedia -vy [tools]
17:11 <arturo> depool tools-exec-1401.tools.eqiad.wmflabs to do some package upgrades [tools]
14:22 <arturo> it was some kind of transient error. After a second puppet run across the fleet, all seems fine [tools]
13:53 <arturo> deploy https://gerrit.wikimedia.org/r/#/c/407465/ which is causing some puppet issues. Investigating. [tools]
2018-02-06 §
13:15 <arturo> deploy https://gerrit.wikimedia.org/r/#/c/408529/ to tools-services-01 [tools]