3201-3250 of 10000 results (36ms)
2021-10-25 §
14:45 <jbond> update cas package [production]
14:41 <majavah> deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image T292771 [toolsbeta]
14:33 <majavah> copy nginx-ingress controller v1.0.4 to internal registry T292771 [tools]
14:31 <marostegui> Deploy schema change on s3 codfw - T291719 [production]
13:02 <btullis> btullis@an-coord1002:~$ sudo systemctl restart hive-server2 hive-metastore [analytics]
12:51 <btullis> btullis@aqs1007:~$ sudo nodetool-a clearsnapshot [analytics]
12:04 <ema> cp3062: upgrade varnish to 6.0.8-1wm2 T293879 [production]
11:57 <ema> deployment-cache-text06: upgrade varnish to 6.0.8-1wm2 T293879 [production]
11:40 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:32 <majavah> depool tools-sgeexec-0910 T294228 [tools]
11:24 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:24 <Lucas_WMDE> UTC morning backport+config window done [production]
11:22 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:732969|Remove dispatchLagToMaxLagFactor Wikibase setting (T292604)]] (duration: 00m 54s) [production]
11:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:18 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:732951|Remove wikibaseDispatchRedisLockManager config (T292604)]] (duration: 00m 54s) [production]
11:14 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732950|Remove wmg variables for dispatchChanges.php Wikibase settings (T292604)]] (duration: 00m 55s) [production]
11:13 <majavah> removed tons of duplicate qw jobs accross multiple tools [tools]
11:10 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:09 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:732949|Remove dispatchChanges.php-related Wikibase settings (T292604)]] (duration: 00m 55s) [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:05 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:732372|Remove dispatchViaJobs-related Wikibase settings (T291828)]] (duration: 00m 56s) [production]
09:52 <godog> bounce uwsgi graphite web on graphite2003 - T294220 [production]
09:52 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:48 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
09:43 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:733089|[BETA CLUSTER] Enable WikibaseLexeme Scribunto access (T294159)]] (merged on Friday, syncing now to avoid outdated files even if it’s just -labs.php) (duration: 00m 55s) [production]
09:18 <godog> bounce graphite-web on graphite2003 to test timeout bump - T294220 [production]
08:08 <XioNoX> merge DNS changes to add drmrs [production]
07:50 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
07:50 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
07:40 <majavah> remove "arturo" from project members and instead add "aborrero" which is the real one [cloudinfra-nfs]
05:47 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,cluster=parsoid,name=wtp1026.* [production]
05:43 <_joe_> pooling wtp1042 T294212 [production]
05:26 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1109.eqiad.wmnet with OS buster [production]
05:01 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1109.eqiad.wmnet with OS buster [production]
04:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1109 (s8) for reimage T290868', diff saved to https://phabricator.wikimedia.org/P17590 and previous config saved to /var/cache/conftool/dbconfig/20211025-043028-marostegui.json [production]
04:26 <Spookyville> Mars instance launched (again) [wikisp]
2021-10-24 §
22:39 <legoktm> one more restart of everything, switched hound container over to bullseye for newer git [codesearch]
22:20 <legoktm> restarted everything to add new prometheus metrics endpoint: https://codesearch.wmcloud.org/_metrics [codesearch]
08:51 <hashar> CI Jenkins: update Clover PHP plugin from 0.5 to 0.6 to solve `java.lang.NoClassDefFoundError: org/apache/commons/digester/Digester` # T292910 [releng]
06:58 <majavah> hard reboot crashed deployment-db08 [releng]
00:47 <andrewbogott> deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738 [admin]
2021-10-23 §
16:40 <dcausse> restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert) [production]
16:06 <majavah> hard reboot codesearch8 after OOM crash [codesearch]
15:45 <urbanecm> Start server-side upload for 1 video file (T289781), testing whether T291137 is still an issue [production]
15:18 <urbanecm> tools.phabbot@tools-sgebastion-07:~$ rm ~/*.err && rm ~/*.out [tools.phabbot]
15:18 <urbanecm> Manually run `tools.phabbot@tools-sgebastion-07:~/phabbot$ ./new_wikis_handler.sh` to re-run the new wikis bot [tools.phabbot]
2021-10-22 §
23:17 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:49 <chicocvenancio> restart to run fix from PR#141 fixes T294150 [tools.video2commons]