51-100 of 10000 results (33ms)
2021-11-05 §
08:34 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Upgrade x2 masters T295026 [production]
08:34 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Upgrade x2 masters T295026 [production]
07:44 <XioNoX> restart scs-a8-eqiad [production]
05:31 <marostegui> Upgrade clouddb1016 [production]
05:31 <marostegui> Upgrade clouddb1020 [production]
00:16 <mutante> phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine [production]
00:06 <mutante> https://labtestwikitech.wikimedia.org - purging mediawiki font packages from backend server [production]
00:04 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
2021-11-04 §
23:51 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:51 <tstarling@deploy1002> Synchronized wmf-config/CommonSettings.php: XWD timeout testing T293568 (duration: 00m 54s) [production]
23:49 <tstarling@deploy1002> Synchronized src/XWikimediaDebug.php: XWD timeout testing (duration: 00m 54s) [production]
23:47 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:44 <cjming> end of UTC late backport & config window [production]
23:44 <cjming@deploy1002> Synchronized wmf-config: Config: [[gerrit:736918|Disable upcoming DiscussionTools mobile interface, enable on beta (T270536)]] (duration: 00m 55s) [production]
23:38 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:736869|Fix value of wgDTSchemaEditAttemptStepSamplingRate (T295052)]] (duration: 00m 55s) [production]
23:37 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:24 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:22 <cjming@deploy1002> Synchronized php-1.38.0-wmf.7/extensions/RelatedArticles: Backport: [[gerrit:736830|Fix loading of related articles via IntersectionObserver (T223844)]] (duration: 00m 55s) [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:19 <mutante> wtp1025, wtp1026, parse2001, parse2002 (parsoid-canary): purging mediawiki font packages (T294378) [production]
23:16 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:736522|Allow bureaucrats to grant and revoke the importer rights to enwikiversity (T294930)]] (duration: 00m 56s) [production]
23:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
21:26 <bblack> cpNNNN: manual (cumin) removal of outdated digicert-2020 ocsp configuration and output files, to avoid icinga alerts and clean up [production]
20:01 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 [production]
20:01 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 [production]
19:29 <dduvall> 1.38.0-wmf.7 on all wikis. no new errors or increase in error rates (refs T293948) [production]
19:25 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:16 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948 [production]
18:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17703 and previous config saved to /var/cache/conftool/dbconfig/20211104-182655-root.json [production]
18:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17701 and previous config saved to /var/cache/conftool/dbconfig/20211104-181151-root.json [production]
18:11 <legoktm> upgrading to scap 4.0.3 on canaries again (T294966) [production]
18:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:08 <legoktm> uploaded scap 4.0.3-2 to apt.wm.o for buster/stretch (T294966) [production]
18:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:06 <jdrewniak@deploy1002> Synchronized portals: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 03s) [production]
18:05 <jdrewniak@deploy1002> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 04s) [production]
17:58 <Amir1> Upgrade db1153 T295026 [production]
17:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 [production]
17:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 [production]
17:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1153 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17700 and previous config saved to /var/cache/conftool/dbconfig/20211104-175606-ladsgroup.json [production]
17:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17699 and previous config saved to /var/cache/conftool/dbconfig/20211104-175429-root.json [production]
17:50 <volans> restarted puppetdb.service on puppetdb2002 [production]
17:47 <ryankemper> T288620 [Elastic] Rebooting `elastic1049.eqiad.wmnet` to uptake new gelf settings change [production]
17:46 <hnowlan> enabling puppet on C:cassandra after profile::java transition [production]
17:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17698 and previous config saved to /var/cache/conftool/dbconfig/20211104-173926-root.json [production]
17:33 <Amir1> Upgrade db1152 T295026 [production]