|
2020-07-09
§
|
| 10:55 |
<_joe_> |
restarting php7.2-fpm on mw1282, workers failing with sigill |
[production] |
| 10:54 |
<_joe_> |
depool mw1282 |
[production] |
| 10:54 |
<mvolz@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:34 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:23 |
<_joe_> |
rolling restart the remaining restbases in eqiad, and all of codfw |
[production] |
| 10:22 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 10:09 |
<_joe_> |
restarting restbase on rb1020-22 |
[production] |
| 09:53 |
<_joe_> |
restarting restbase on restbase1024,1023 |
[production] |
| 09:36 |
<_joe_> |
restarting restbase on rb1026,1027 to switch to proton on k8s |
[production] |
| 09:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:28 |
<_joe_> |
restarting restbase on restbase1025 to pick up the switch to k8s of proton |
[production] |
| 09:27 |
<godog> |
bounce thanos-compact on thanos-fe2001 |
[production] |
| 09:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0) |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P11828 and previous config saved to /var/cache/conftool/dbconfig/20200709-085228-marostegui.json |
[production] |
| 08:44 |
<marostegui> |
Stop haproxy on dbproxy1017 before upgrading to buster - T255408 |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1136', diff saved to https://phabricator.wikimedia.org/P11827 and previous config saved to /var/cache/conftool/dbconfig/20200709-082355-marostegui.json |
[production] |
| 08:23 |
<moritzm> |
imported osm2pgsql 0.96.0+ds-1~bpo9+1 to "main" component T256877 |
[production] |
| 08:22 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 08:20 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 08:13 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 08:11 |
<XioNoX> |
disable igmp snooping on msw1-codfw |
[production] |
| 07:59 |
<marostegui> |
Stop db1117:3322 to clone db1084, this will trigger haproxy alerts - T257540 |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11825 and previous config saved to /var/cache/conftool/dbconfig/20200709-075749-marostegui.json |
[production] |
| 07:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11824 and previous config saved to /var/cache/conftool/dbconfig/20200709-053905-marostegui.json |
[production] |
| 05:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1084 from dbctl', diff saved to https://phabricator.wikimedia.org/P11823 and previous config saved to /var/cache/conftool/dbconfig/20200709-053206-marostegui.json |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084', diff saved to https://phabricator.wikimedia.org/P11822 and previous config saved to /var/cache/conftool/dbconfig/20200709-051826-marostegui.json |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11821 and previous config saved to /var/cache/conftool/dbconfig/20200709-051355-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Remove revision triggers from db2093:3315 T238966 |
[production] |
| 05:10 |
<marostegui> |
Deploy schema change on s5 codfw, lag will be generated - T238966 |
[production] |
| 01:43 |
<tzatziki> |
reset email for GseSro |
[production] |
| 00:58 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"' |
[production] |
| 00:49 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'disable-puppet "cdanis deploying I6c1b646e T256395"' |
[production] |
|
2020-07-08
§
|
| 21:56 |
<mutante> |
deleting files from releases2001 that are not existing on releases1001 to make them mirrors. rsync with --delete and the command from quickdatacopy class (T247652) |
[production] |
| 21:55 |
<mutante> |
rsyncing releases files from releases1001 to releases2002 and releases1002. deleting files from releases2002 not existing on releases1002 to make them mirrors ( T247652_ |
[production] |
| 20:59 |
<cstone> |
civicrm revision changed from d73ee2e73f to 8b09c87ce2, |
[production] |
| 20:27 |
<Amir1> |
end of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T256012) |
[production] |
| 20:08 |
<Amir1_> |
start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T256012) |
[production] |
| 19:18 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.40 refs T256668 (duration: 01m 04s) |
[production] |
| 19:17 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.40 refs T256668 |
[production] |
| 18:58 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 091442cf035a6d76f1211291afbb3193c513595d: Add *.nga.gov to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T256518) (duration: 01m 04s) |
[production] |
| 18:55 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 2e5943ddb30e08607a9ffb6ed05a042e8367e2e1: Add scan-bugs.org to $wgCopyUploadsDomains (T256569) (duration: 01m 04s) |
[production] |
| 18:46 |
<urbanecm@deploy1001> |
Synchronized static/images/project-logos/: f42cdf2: Change bnwiki logo (T255328) (duration: 01m 04s) |
[production] |
| 18:27 |
<ppchelko@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Cleanup: remove temporary wmgDisableHTCP variable gerrit:607596 T250781 IS.php (duration: 01m 01s) |
[production] |
| 18:20 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Disable HTCP purging everywhere gerrit:607593 T250781 CS.php (duration: 01m 03s) |
[production] |
| 18:18 |
<ppchelko@deploy1001> |
Synchronized wmf-config/wikitech.php: Disable HTCP purging everywhere gerrit:607593 T250781 wikitech.php (duration: 01m 04s) |
[production] |