7701-7750 of 10000 results (50ms)
2021-09-08 §
07:45 <godog> start rollout of prometheus-rsyslog-exporter 0.0.0+git20201008-3 to eqsin/esams/ulsfo - T210137 [production]
06:46 <ryankemper> [WDQS] Manually running puppet-agent on `miscweb2002.codfw.wmnet,miscweb1002.eqiad.wmnet` [production]
06:45 <ryankemper> [WDQS] Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/719185 to rollback query.wikidata.org changes [production]
02:59 <eileen> civicrm revision changed from 06ef98593f to 593d01f4fc, config revision is 5f004d94d7 [production]
00:00 <legoktm> legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303 [production]
2021-09-07 §
23:25 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:20 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
23:13 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:719381|Enable UrlShortener everywhere (T267925)]] (duration: 00m 58s) [production]
23:07 <dpifke@deploy1002> Synchronized wmf-config/profiler.php: Config: [[gerrit:716041|profiler: use seperate pipeline inside k8s pods (T288165)]] (duration: 00m 58s) [production]
22:29 <cstone> SmashPig revision changed from afd362b163 to 3607b16f83 [production]
20:41 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:715018|Set $wgWBRepoSettings['tmpNormalizeDataValues'] on all wikis (T251480)]] (duration: 00m 59s) [production]
20:31 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:27 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
17:18 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
17:09 <jgiannelos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
17:01 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
16:39 <moritzm> installing jetty9 security updates on buster [production]
16:30 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue [production]
16:30 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue [production]
16:30 <dancy@deploy1002> Synchronized README: testing (duration: 00m 59s) [production]
15:18 <akosiaris> run_benchmarky.py against mwdebug.svc.codfw.wmnet for performance tests [production]
15:07 <akosiaris@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:04 <jbond> upload python-prometheus-client_0.6.0 to stretch-wikimedia [production]
14:50 <mutante> snapshot1015 - manually removed prometheus-puppet-agent-stats from crontab which was sending spam and is now a timer [production]
14:33 <mutante> CI - migrating zuul-merger cronjob to systemd timer (contint*) [production]
14:23 <XioNoX> re-pool esams-eqiad - T288503 [production]
14:23 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1024.eqiad.wmnet with reason: REIMAGE [production]
14:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1024.eqiad.wmnet with reason: REIMAGE [production]
14:22 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1023.eqiad.wmnet with reason: REIMAGE [production]
14:22 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1023.eqiad.wmnet with reason: REIMAGE [production]
14:17 <marostegui> No more db maintenance on eqiad T288594 [production]
14:08 <mutante> alert1001 - temp disabled puppet, stopped icinga-wm [production]
14:07 <mutante> temp killed icinga-wm because of flooding [production]
14:01 <Emperor> removing pc2010 from orchestrator T289117 [production]
13:59 <Emperor> removing pc2010 from tendril and zarcillo T289117 [production]
13:57 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:57 <XioNoX> drain esams-eqiad for circuit maintenance - T288503 [production]
13:54 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
13:51 <jayme> uncordoned kubestage2001 [production]
13:50 <jiji@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:49 <mutante> mw2264 - scap pulled and repooled after T290242 [production]
13:49 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2264.codfw.wmnet [production]
13:43 <jiji@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:40 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc2010.codfw.wmnet [production]
13:25 <mvernon@cumin1001> START - Cookbook sre.hosts.decommission for hosts pc2010.codfw.wmnet [production]
13:21 <Emperor> removing pc2009 from orchestrator T289116 [production]
13:21 <Emperor> removing pc2009 from tendril and zarcillo T289116 [production]
13:02 <marostegui@cumin1001> dbctl commit (dc=all): 'fix s8 weights T288594', diff saved to https://phabricator.wikimedia.org/P17248 and previous config saved to /var/cache/conftool/dbconfig/20210907-130244-marostegui.json [production]
12:59 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc2009.codfw.wmnet [production]
12:51 <mvernon@deploy1002> Synchronized wmf-config/ProductionServices.php: Remove old decommissioned pc hosts T284825 (duration: 01m 02s) [production]