|
2020-02-19
ยง
|
| 21:23 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:23 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:55 |
<eileen> |
civicrm revision changed from 52c68911c6 to a6b222c19f, config revision is 561ae21f77 |
[production] |
| 20:15 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.20/extensions/Wikibase/lib: Fix stastd metric for StatsdMissRecordingSimpleCache (wb_terms work) (duration: 01m 06s) |
[production] |
| 20:13 |
<rzl@cumin1001> |
conftool action : set/weight=30; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet |
[production] |
| 20:12 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.19/extensions/Wikibase/lib: Fix stastd metric for StatsdMissRecordingSimpleCache (wb_terms work) (duration: 01m 06s) |
[production] |
| 20:10 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.19/extensions/Wikibase/lib: Fix stastd metric for StatsdMissRecordingSimpleCache (wb_terms work) (duration: 01m 05s) |
[production] |
| 20:05 |
<jforrester@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.20 (duration: 01m 03s) |
[production] |
| 20:04 |
<jforrester@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.20 |
[production] |
| 20:02 |
<rzl@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet |
[production] |
| 20:02 |
<rzl@cumin1001> |
conftool action : set/weight=10; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet |
[production] |
| 19:54 |
<rlazarus> |
scap pull on new api servers mw13[56-62] |
[production] |
| 19:50 |
<mutante> |
generating mcrouter certs for new codfw mw appservers |
[production] |
| 19:39 |
<mutante> |
initial puppet run on new hosts mw231* |
[production] |
| 19:31 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.19/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s) |
[production] |
| 19:27 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.20/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s) |
[production] |
| 19:21 |
<jforrester@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: T244577 [metawiki] Disable MobileFrontend mainpage special casing (duration: 01m 04s) |
[production] |
| 19:18 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T244369 [trwiki] Enable the WikidataPageBanner extension (duration: 01m 05s) |
[production] |
| 19:11 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.20/includes/resourceloader/dependencystore/SqlModuleDependencyStore.php: T245570 resourceloader: fix SqlDependencyModuleStore::setMulti() to use upsert() (duration: 01m 01s) |
[production] |
| 18:45 |
<bblack> |
dns4001 - upgraded to gdnsd-3.2.2 |
[production] |
| 18:44 |
<bblack> |
reprepro: upload gdnsd 3.2.2-1~wmf1 to buster-wikimedia |
[production] |
| 18:39 |
<mutante> |
mwmaint1002 - sudo systemctl reset-failed to clear systemd alerts |
[production] |
| 18:38 |
<mutante> |
mwmaint1002 - removing Icinga ACK for systemd state - comments for it were from HHVM removal in Oct 2019 |
[production] |
| 18:26 |
<mutante> |
phab2001 - upgraded ssh-server, kept locally modified config; apt autoremove removes python3-debconf |
[production] |
| 18:23 |
<mutante> |
phab2001 - installing package upgrades, incl. openssh, PHP version |
[production] |
| 18:22 |
<mutante> |
phab2001 - upgrading mariadb client package versions |
[production] |
| 18:19 |
<mutante> |
removing problem ACK from Icinga alerts for wikitech-static MediaWiki version. comments were about things in 2019 |
[production] |
| 17:48 |
<robh> |
cp1089 cp1090 returned to service via T243167 |
[production] |
| 17:40 |
<jynus> |
starting data check between db1078 and db1140:3313 T244958 |
[production] |
| 17:39 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q4000 (T225057) (just incase of cache issue) (duration: 01m 04s) |
[production] |
| 17:26 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q4000 (T225057) (duration: 01m 01s) |
[production] |
| 17:14 |
<ema> |
cp4026: repool after probe Connection:keep-alive experiment revert https://gerrit.wikimedia.org/r/573337 |
[production] |
| 17:12 |
<robh> |
cp1088 returned to service, cp1089 & cp1090 offline for firmware update via T243167 |
[production] |
| 16:44 |
<papaul> |
replacing ps1-a8-codfw mgmt in rack A8 will go down |
[production] |
| 16:37 |
<otto@deploy1001> |
Finished deploy [analytics/refinery@e23918a]: Updating eventgate-analytics port (T245203) and also eventlogging whitelist (duration: 12m 27s) |
[production] |
| 16:32 |
<ema> |
depool cp4026, 5xx |
[production] |
| 16:24 |
<otto@deploy1001> |
Started deploy [analytics/refinery@e23918a]: Updating eventgate-analytics port (T245203) and also eventlogging whitelist |
[production] |
| 16:13 |
<marostegui> |
Depool labsdb1011 to help replication to catch up |
[production] |
| 16:05 |
<elukey> |
Update analytics-in4 filter term eventgate for T245203 on cr1/cr2 eqiad |
[production] |
| 15:48 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@b42acb5]: fix temp stub generation, add pagerangeinfo cache, some unit tests (duration: 00m 03s) |
[production] |
| 15:48 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@b42acb5]: fix temp stub generation, add pagerangeinfo cache, some unit tests |
[production] |
| 14:59 |
<marostegui> |
Stop mysql on es2021 - T243052 |
[production] |
| 14:31 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:29 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:29 |
<marostegui> |
Data checksum on db1084 T245621 |
[production] |
| 14:07 |
<marostegui> |
Upgrade and reboot db1084 - T245621 |
[production] |
| 14:02 |
<marostegui> |
Start mysql on db1084 without replication - T245621 |
[production] |
| 13:53 |
<jbond42> |
disable puppet to upgrade postgresql |
[production] |
| 13:30 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1084, lots of connection errors', diff saved to https://phabricator.wikimedia.org/P10458 and previous config saved to /var/cache/conftool/dbconfig/20200219-133057-jynus.json |
[production] |
| 12:25 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573236|Start reading for the new term store for clients up to Q2000 (T225057)]], take II, the cache issue (duration: 01m 04s) |
[production] |