|
2021-03-11
ยง
|
| 21:20 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker |
[production] |
| 21:17 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:15 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:14 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:670858|Enable GrowthExperiments link recommendations on testwiki (T277173)] (duration: 00m 59s) |
[production] |
| 21:13 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s) |
[production] |
| 21:12 |
<bd808> |
Update demo server with search and toolinfo creation features |
[toolhub] |
| 21:12 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date |
[production] |
| 21:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2216.codfw.wmnet |
[production] |
| 21:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw2215.codfw.wmnet |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:58 |
<mutante> |
deactivating codfw API canaries on old hardware (T277119) |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2216.codfw.wmnet |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2215.codfw.wmnet |
[production] |
| 20:50 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:46 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s) |
[production] |
| 20:44 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment |
[production] |
| 20:35 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:33 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:28 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:20 |
<razzi> |
disable maintenance mode for matomo1002 |
[analytics] |
| 20:20 |
<mutante> |
phab1001 - systemctl start phabricator_clean_tmp_files - now Succeeded |
[production] |
| 20:17 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1002.eqiad.wmnet |
[production] |
| 20:13 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host matomo1002.eqiad.wmnet |
[production] |
| 20:08 |
<razzi> |
starting reboot of matomo1002 for kernel upgrade |
[analytics] |
| 20:04 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 |
[production] |
| 19:59 |
<mutante> |
phab1001 - sudo systemctl start phabricator_clean_tmp_files (manually run after conversion from cron to timer, and it fails with permission issues) |
[production] |
| 19:55 |
<tgr_> |
T277173 running mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=testwiki GrowthExperiments |
[production] |
| 19:54 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670857|Configure GrowthExperiments Add Link settings, step 2 (T277173)]] (duration: 01m 08s) |
[production] |
| 19:43 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:30 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670887|Configure GrowthExperiments Add Link settings, step 1 (T277173)]] (duration: 01m 08s) |
[production] |
| 19:18 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:668196|wikitech: enable BetaFeatures (T125941)]] (duration: 01m 08s) |
[production] |
| 19:13 |
<hnowlan@deploy1002> |
Finished deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues (duration: 16m 25s) |
[production] |
| 18:56 |
<hnowlan@deploy1002> |
Started deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues |
[production] |
| 18:52 |
<razzi> |
systemctl restart hadoop-hdfs-datanode on analytics1059 |
[analytics] |
| 18:50 |
<razzi> |
systemctl restart hadoop-yarn-nodemanager on analytics1059 |
[analytics] |
| 18:47 |
<tgr_> |
running mwscript extensions/GrowthExperiments/maintenance/importOresTopics.php testwiki --count 1000 --verbose --wikiId enwiki --apiUrl 'https://en.wikipedia.org/w/api.php' |
[production] |
| 18:38 |
<legoktm> |
upgrade pip in virtualenv and downgrade pymysql to 0.10 |
[tools.tutor] |
| 18:35 |
<razzi> |
apt-get install parted on analytics1059 |
[analytics] |
| 18:33 |
<bstorm> |
silenced alerts from deploymentprep for another 60 days |
[metricsinfra] |
| 17:40 |
<bstorm> |
deployed metrics-server:0.4.1 to kubernetes |
[tools] |
| 17:31 |
<effie> |
install mecached 1.6.6-1 on mwdebug1001 |
[production] |
| 16:57 |
<Majavah> |
copy a tarball of deployment-fluorine02 /home to deployment-mwlog01 root home dir, delete deployment-fluorine02 T276419 |
[releng] |