|
2021-07-29
§
|
| 12:22 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti2026.codfw.wmnet |
[production] |
| 12:21 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet |
[production] |
| 10:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2104 T287230', diff saved to https://phabricator.wikimedia.org/P16925 and previous config saved to /var/cache/conftool/dbconfig/20210729-102753-marostegui.json |
[production] |
| 09:59 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola (duration: 00m 22s) |
[production] |
| 09:59 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola |
[production] |
| 09:47 |
<moritzm> |
installing Mariadb 10.3.29 updates from Buster point release (as packaged in Debian, not the WMF DB packages) |
[production] |
| 09:40 |
<jelto> |
uncordon kubestage1002.eqiad.wmnet as rsyslog was restarted and log shipping to logstash works again |
[production] |
| 09:14 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet |
[production] |
| 09:04 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet |
[production] |
| 08:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 08:51 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:33 |
<moritzm> |
purging obsolete kernels from moscovium (disk space alerts for /) |
[production] |
| 08:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moscovium.eqiad.wmnet |
[production] |
| 08:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host moscovium.eqiad.wmnet |
[production] |
| 07:55 |
<elukey> |
roll restart uwsgi + celery on ores[12]* nodes to pick up aspell upgrades |
[production] |
| 07:53 |
<mbsantos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 07:52 |
<moritzm> |
restarting Tomcat on idp-test |
[production] |
| 06:41 |
<XioNoX> |
push pfw policies - T287203 |
[production] |
| 05:44 |
<Amir1> |
adding "comunicaciones AT wikimediacolombia.org" as owner of wikimedia-co mailing list |
[production] |
| 01:08 |
<eileen> |
civicrm revision changed from 739c936298 to 158ed65e00, config revision is 6011d9c471 |
[production] |
|
2021-07-28
§
|
| 23:57 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:708581|wgSkipSkins: Update defaults, hide modern (T287616)]] (duration: 01m 06s) |
[production] |
| 23:50 |
<thcipriani@deploy1002> |
Synchronized wmf-config: Config: [[gerrit:708158|Disable mobile contributions simplifications on Wikidata and Commons (T283988)]] (duration: 01m 58s) |
[production] |
| 21:33 |
<majavah> |
add mdipietro as projectadmin and to sudo policy T287287 |
[tools] |
| 21:29 |
<majavah> |
add mdipietro as maintainer T287287 |
[tools.admin] |
| 19:16 |
<twentyafterfour@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.16 refs T281157 (duration: 01m 06s) |
[production] |
| 19:15 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.16 refs T281157 |
[production] |
| 19:09 |
<twentyafterfour> |
Preparing to deploy 1.37.0-wmf.16 to group1 wikis |
[production] |
| 18:57 |
<legoktm> |
mwmaint2002$ foreachwikiindblist wikimania refreshLinks.php - to start populating DPL tracking category |
[production] |
| 18:36 |
<legoktm@deploy1002> |
Finished scap: Add a tracking category to pages using the <DynamicPageList> tag (duration: 27m 16s) |
[production] |
| 18:14 |
<jbond> |
manually cleared out the puppetdb2002 queue |
[production] |
| 18:08 |
<legoktm@deploy1002> |
Started scap: Add a tracking category to pages using the <DynamicPageList> tag |
[production] |
| 17:53 |
<andrewbogott> |
rebooting deployment-logstash03 as it's in an inconsistent config state |
[deployment-prep] |
| 17:53 |
<andrewbogott> |
rebooting deployment-logstash03 as it's in an inconsistent config state |
[releng] |
| 17:14 |
<bd808> |
mwv-builder-03: added cinder volume, migrated /srv to it, fixed broken apt state, forced puppet run to ensure things are better now that the / partition has some free space. |
[mediawiki-vagrant] |
| 16:59 |
<bd808> |
Adding cinder volume to hold /srv data. |
[mediawiki-vagrant] |
| 16:54 |
<andrewbogott> |
rebooting relforge-search; it's unreachable and hanging |
[search] |
| 16:52 |
<andrewbogott> |
rebooting VM 'backend', unreachable and hanging |
[wikicommunityhealth] |
| 16:49 |
<bd808> |
Hard reboot mwv-builder-03. Unresponsive to ssh and http. |
[mediawiki-vagrant] |
| 16:47 |
<andrewbogott> |
rebooting nehpets VM, OOM |
[reading-web-staging] |
| 16:39 |
<andrewbogott> |
rebooting gerrit-prod-1001; seemingly unreachable |
[devtools] |
| 16:37 |
<ryankemper> |
[WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good |
[production] |
| 16:18 |
<wm-bot> |
` sudo docker restart 873f4b18478d` Restarting wikibase-registry_wdqs-frontend_1 to hopefully fix all wdqs queries being 502 |
[wikibase-registry] |
| 16:17 |
<wm-bot> |
` sudo docker restart 6e997bf4a59e` Restarting wikibase-registry_wdqs-proxy_1 to hopefully fix all wdqs queries being 502 |
[wikibase-registry] |
| 16:11 |
<wm-bot> |
` sudo docker restart b6f6d2d0dd7a` Restarting wikibase-registry_wdqs-0310_1 to hopefully fix all wdqs queries being 502 |
[wikibase-registry] |
| 16:00 |
<ryankemper> |
[WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` |
[production] |
| 15:59 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` |
[production] |
| 15:59 |
<ryankemper> |
[WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` |
[production] |
| 15:58 |
<ryankemper> |
T287112 [WDQS] Re-pooled `wdqs2002` |
[production] |