|
2019-10-04
§
|
| 22:06 |
<mutante> |
ms-be1020 - power cycle via mgmt - host down |
[production] |
| 21:43 |
<bd808> |
`sudo exec-manage repool tools-sgeexec-0923.tools.eqiad.wmflabs` |
[tools] |
| 21:26 |
<bd808> |
Rebooting tools-sgeexec-0923 after lots of messing about with a broken update-initramfs build |
[tools] |
| 21:13 |
<hauskater> |
gerrit: Ran replication start mediawiki/extensions/WikiSEO --wait # New repository |
[releng] |
| 20:43 |
<krinkle@deploy1001> |
Synchronized w/static.php: 9648e03, 97d9384 (duration: 00m 53s) |
[production] |
| 20:41 |
<mutante> |
deploy1001 / deploy2001 - remove python-pygerrit2 (version for python3 is needed instead) |
[production] |
| 20:35 |
<bd808> |
Manually running `/usr/bin/python3 /usr/bin/unattended-upgrade` on tools-sgeexec-0923 |
[tools] |
| 20:33 |
<bd808> |
Killed 2 /usr/bin/unattended-upgrade procs on tools-sgeexec-0923 that seemed stuck |
[tools] |
| 20:32 |
<mutante> |
gerrit1001 - scp /usr/share/java/mysql-connector-java.jar from cobalt into /usr/share/java/ on gerrit1001 and then symlink into /var/lib/gerrit2/review_site/lib/ (T222391) |
[production] |
| 19:34 |
<marxarelli> |
re-running `mediawiki/tools/release/make-release/branch.py --bundle '*' ...` due to https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/540953 cc: T232024 |
[releng] |
| 19:27 |
<mutante> |
wtp1025 - mediawiki appserver classes are being applied, install in progress will trigger some new icinga alerts |
[production] |
| 19:17 |
<Krinkle> |
Fix Jenkins Console section "setup quibble prepare" pattern to account for recent changes on "mwext-php72-phan-docker" which was eating the pattern incorrectly |
[releng] |
| 18:55 |
<marxarelli> |
branching REL1_34 for remaining extensions/skins and core, `mediawiki/tools/release/make-release/branch.py REL1_34 --bundle '*' --core --branchpoint master --core-version 1.34.0` (note: core branchpoint is overridden in settings.yaml at base of wmf/1.34.0-wmf.25) cc: T232024 |
[releng] |
| 18:34 |
<marxarelli> |
branching REL1_34 for all wmf-deployed extensions, `mediawiki/tools/release/make-release/branch.py REL1_34 --bundle wmf --branchpoint wmf/1.34.0-wmf.25 --core-version 1.34.0` cc: T232024 |
[releng] |
| 17:45 |
<brennen> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/540699 for parsoid publication pipeline |
[releng] |
| 16:27 |
<ottomata> |
manually rsyncing mediawiki_history 2019-08 snapshot to labstore1006 |
[analytics] |
| 15:00 |
<Zppix> |
Reset password for user Halfak per IRC request |
[git] |
| 14:03 |
<marostegui> |
Deploy schema change on db2117 T233135 T234066 |
[production] |
| 13:50 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'restrouter' for release 'production' . |
[production] |
| 13:47 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'restrouter' for release 'production' . |
[production] |
| 13:36 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'restrouter' for release 'staging' . |
[production] |
| 13:33 |
<arturo> |
remove /etc/init.d/rsyslog on tools-worker-XXXX nodes so the rsyslog deb prerm script doesn't prevent the package from being updated |
[tools] |
| 12:28 |
<marostegui> |
Deploy schema change on db2097:3316 T233135 T234066 |
[production] |
| 12:23 |
<elukey> |
cleaned up old files and apt-cache from an-coord1001 |
[production] |
| 08:41 |
<marostegui> |
Deploy schema change on db2076 (sanitarium master) with replication T233135 T234066 |
[production] |
| 08:32 |
<_joe_> |
reuploading the old confd package to stetch-wikimedia, some incompatibility detected |
[production] |
| 07:44 |
<hashar> |
Adding labels to Jenkins agent "Docker blubber m4executor" on integration-agent-jessie-docker-1001 | Jessie based VM to potentially compare speed of jobs with the Stretch based ones |
[releng] |
| 07:26 |
<elukey> |
execute gnt-instance remove kerberos1001 on ganeti1001 - T234600 |
[production] |
| 07:24 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 07:24 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:40 |
<marostegui> |
Deploy schema change on db2114 T233135 T234066 |
[production] |
| 06:22 |
<_joe_> |
downgrading confd back to 0.9.0 while some templates get fixed. |
[production] |
| 06:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:16 |
<marostegui> |
Deploy schema change on dbstore1005:3316 T233135 T234066 |
[production] |
| 05:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool es1019 after on-site maintenance T233698 (duration: 00m 51s) |
[production] |
| 05:53 |
<_joe_> |
upgrading confd on puppetmaster1001 T147204 |
[production] |
| 05:50 |
<_joe_> |
uploading confd 0.16.0 on stretch T147204 |
[production] |
| 05:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to es1019 after on-site maintenance T233698 (duration: 00m 51s) |
[production] |
| 05:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P9240 and previous config saved to /var/cache/conftool/dbconfig/20191004-051112-marostegui.json |
[production] |
| 05:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool es1019 after on-site maintenance T233698 (duration: 00m 53s) |
[production] |