|
2017-11-20
§
|
| 10:19 |
<dcausse> |
elastic/cirrus: reindexing english group0 and group1 wikis: T179945 |
[production] |
| 10:08 |
<hashar> |
contint1001: sudo systemctl start jenkins |
[production] |
| 10:06 |
<hashar> |
nodepool: manually deleted left over instances ci-jessie-wikimedia-894187 and ci-jessie-wikimedia-894188 . Jenkins fails to ssh to it and they were left ready for 72 hours. |
[releng] |
| 10:05 |
<hashar> |
deployment-phab : set hiera 'phabricator_cluster_search: []' trying to unblock puppet and soft rebooted the instance | T180935 |
[releng] |
| 10:05 |
<moritzm> |
rebooting contint1001 for update to 4.9.51 |
[production] |
| 09:39 |
<hashar> |
deployment-prep added missing key between_bytes_timeout to cache::app_def_be_opts for deployment-cache-text04 and deployment-cache-upload04 | T180935 |
[releng] |
| 09:29 |
<hashar> |
deployment-tin: apt-mark hold scap | the apt-repo on deployment-tin is out of date | T180935 |
[releng] |
| 09:22 |
<moritzm> |
rebooting hafnium for update to 4.9.51 |
[production] |
| 08:46 |
<marostegui> |
Run mydumper for db1047.staging - T156844 |
[production] |
| 07:58 |
<moritzm> |
installing procmail security updates |
[production] |
| 07:39 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1097 - T178359 (duration: 00m 49s) |
[production] |
| 06:41 |
<marostegui> |
Stop MySQL on db1082 to clone db1109 and db1110 - T180700 |
[production] |
| 06:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1082 - T177208 (duration: 00m 48s) |
[production] |
| 06:25 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore original weights for db1100 and db1071 - T180917 (duration: 00m 49s) |
[production] |
| 06:15 |
<marostegui> |
Reboot db2068 - T180927 |
[production] |
| 02:28 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.7) (duration: 06m 26s) |
[production] |
| 00:52 |
<andrewbogott> |
cherry-picking https://gerrit.wikimedia.org/r/#/c/392172/ onto the tools puppetmaster |
[tools] |
|
2017-11-18
§
|
| 23:25 |
<madhuvishy> |
killed pid 25631 - /usr/lib/gridengine/qacct -j 2067313 |
[tools.ukbot] |
| 21:40 |
<Krinkle> |
Deleting cvn-apache8 (replaced by cvn-apache9) |
[cvn] |
| 20:06 |
<gehel> |
upgrade of elasticsearch eqiad complete - T178411 |
[production] |
| 18:03 |
<reedy@tin> |
Synchronized multiversion/vendor/: bump (duration: 01m 14s) |
[production] |
| 15:53 |
<marostegui> |
Compress s5 on db2085 - T178359 |
[production] |
| 14:48 |
<marostegui> |
Compress s3 on db2092 - T178359 |
[production] |
| 02:36 |
<Krinkle> |
Shutdown now-unused cvn-apache8 instance. To be deleted later. |
[cvn] |
| 02:36 |
<Krinkle> |
Switch cvn.wmflabs.org web proxy over to new cvn-apache9 backend. |
[cvn] |
| 02:14 |
<Krinkle> |
Create and provision cvn-apache9 (Debian 9 Stretch) to replace cvn-apache8 (Ubuntu 14 Trusty) |
[cvn] |
| 02:11 |
<zhuyifei1999_> |
Removing myself from maintainer list (cc T179599) |
[tools.tb-dev] |
| 00:30 |
<urandom> |
Bootstrapping restbase2006-b (T179422) |
[production] |
|
2017-11-17
§
|
| 23:17 |
<Sagan> |
CAC: Updating packages and running autoremove |
[rcm] |
| 23:15 |
<Sagan> |
CAC: Fixing errors from last ran via foreachwiki update.php --quick |
[rcm] |
| 23:15 |
<Sagan> |
CAC: Running composer update on vagrant |
[rcm] |
| 23:12 |
<Sagan> |
Xenon: Doing package updates and autoremove |
[rcm] |
| 23:12 |
<Sagan> |
Neon: Updating packages |
[rcm] |
| 23:11 |
<Sagan> |
CAC: Updating vagrant, including vagrant git-update |
[rcm] |
| 23:10 |
<Sagan> |
Oxygen: Updateing packages |
[rcm] |
| 23:08 |
<Sagan> |
Tin: Updating packages |
[rcm] |
| 23:06 |
<Sagan> |
Xenon: Running update |
[rcm] |
| 21:33 |
<valhallasw`cloud> |
also g-w'ed those files, and sent emails to all the affected users |
[tools] |
| 21:31 |
<JeanFred> |
Reverted to old database replicas (via `git reset HEAD~1 && git stash`) as part of T180833 investigation |
[tools.heritage] |
| 21:17 |
<valhallasw`cloud> |
chmod o-w'ed a bunch of files reported by Dispenser; writing emails to the owners about this |
[tools] |
| 20:35 |
<mutante> |
netmon1002 - rsync smokeping data back from local backup to show measurements made from eqiad as requested on T180812 |
[production] |
| 20:05 |
<mutante> |
running puppet on all cache misc to switch smokeping web to eqiad |
[production] |
| 20:02 |
<mutante> |
T180812 copying smokeping data from 2001 to 1002 - netmon1002: /usr/bin/rsync -avp rsync://netmon2001.wikimedia.org/var-lib-smokeping /var/lib/smokeping/ | switching backend from codfw to eqiad |
[production] |
| 19:37 |
<mutante> |
T180812 - @netmon1002:# rsync -avp /var/lib/smokeping/ /root/backup/netmon1002/201711717/var/lib/smokeping/@netmon2001:/# rsync -avp /var/lib/smokeping/ /root/backup/netmon2001/201711717/var/lib/smokeping/ |
[production] |