|
2022-03-17
§
|
| 06:57 |
<ryankemper> |
[WDQS] Note that per https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&var-cluster_name=wdqs&from=1647457172391&to=1647500081971&viewPanel=7 `wdqs2003` has been offline for ~6 hours, `wdqs2001` for 1.5 hours and `wdqs2004` just recently. |
[production] |
| 06:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 75%: After buffer pool testing', diff saved to https://phabricator.wikimedia.org/P22741 and previous config saved to /var/cache/conftool/dbconfig/20220317-065656-root.json |
[production] |
| 06:54 |
<ryankemper> |
[WDQS] `ryankemper@wdqs2003:~$ sudo systemctl restart wdqs-blazegraph.service` |
[production] |
| 06:53 |
<ryankemper> |
[WDQS] `ryankemper@wdqs2001:~$ sudo systemctl restart wdqs-blazegraph.service` |
[production] |
| 06:50 |
<elukey> |
restart blazegraph on wdqs2004 |
[production] |
| 06:46 |
<elukey> |
kill remaining hanging processes for ppche*lko and accra*ze on an-test-client1001 to allow users offboard (puppet broken) |
[analytics] |
| 06:46 |
<elukey> |
kill remaining hanging processes for ppche*lko and accra*ze on an-test-client1001 to allow users offboard (puppet broken) |
[production] |
| 06:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 50%: After buffer pool testing', diff saved to https://phabricator.wikimedia.org/P22740 and previous config saved to /var/cache/conftool/dbconfig/20220317-064152-root.json |
[production] |
| 06:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 25%: After buffer pool testing', diff saved to https://phabricator.wikimedia.org/P22739 and previous config saved to /var/cache/conftool/dbconfig/20220317-062648-root.json |
[production] |
| 06:15 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 10%: After buffer pool testing', diff saved to https://phabricator.wikimedia.org/P22738 and previous config saved to /var/cache/conftool/dbconfig/20220317-061144-root.json |
[production] |
| 04:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1146:3314 (T300775)', diff saved to https://phabricator.wikimedia.org/P22737 and previous config saved to /var/cache/conftool/dbconfig/20220317-040634-marostegui.json |
[production] |
| 04:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:06 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:57 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1016.eqiad.wmnet with OS bullseye |
[production] |
| 02:07 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye |
[production] |
| 02:07 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1016.eqiad.wmnet with OS bullseye |
[production] |
| 01:11 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye |
[production] |
| 01:09 |
<wm-bot> |
Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster |
[admin] |
| 00:53 |
<wm-bot> |
Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster |
[admin] |
| 00:52 |
<wm-bot> |
Setting cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster |
[admin] |
| 00:52 |
<wm-bot> |
Draining 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster |
[admin] |
| 00:44 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[pipelinelib-experimental] |
| 00:43 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[wikidata-realtime-dumps] |
| 00:42 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[wikidata-autodesc] |
| 00:40 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[thumbor] |
| 00:39 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[sentry] |
| 00:38 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[redwarn] |
| 00:37 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[push] |
| 00:36 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[privpol-captcha] |
| 00:33 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[openrefine] |
| 00:30 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[globalcu] |
| 00:29 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[glampipe] |
| 00:28 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[community-labs-monitoring] |
| 00:27 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[antiharassment] |
| 00:21 |
<andrewbogott> |
deleting remaining VMs and project, as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2021_Purge |
[annotation] |
|
2022-03-16
§
|
| 23:52 |
<tzatziki> |
Removing two files for legal compliance |
[production] |
| 22:00 |
<James_F> |
Docker: Publishing sonar-scanner:4.6.0.2311-3 for T303958 |
[releng] |
| 21:17 |
<cjming> |
end running skin update preference maintenance script |
[production] |
| 20:52 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:40 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: [no-op] 8efa537: GrowthExperiments: Set GEWelcomeSurveyShowMailingListQuestion (T303240) (duration: 00m 53s) |
[production] |
| 20:38 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:35 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.26/extensions/WikimediaMaintenance/: 9ba157b: Add insert option for update skin preferences script (T299104) (duration: 00m 50s) |
[production] |
| 20:34 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.25/extensions/WikimediaMaintenance/: ebfc516: Add script to update vector skin preferences (T299104) (duration: 00m 51s) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:24 |
<pt1979@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1025.eqiad.wmnet with OS bullseye |
[production] |
| 20:13 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 20:13 |
<James_F> |
Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for T302607 and T302608 |
[releng] |
| 20:13 |
<urbanecm@deploy1002> |
Synchronized docroot/noc/db.php: f649199: Migrate wmfDatacenter(s) to wmgDatacenter(s) (T45956; 3/3) (duration: 00m 49s) |
[production] |