|
2021-06-07
§
|
| 22:02 |
<urbanecm> |
urbanecm@deployment-sessionstore04:~$ sudo service cassandra start # T263617 |
[releng] |
| 22:02 |
<urbanecm> |
urbanecm@deployment-sessionstore04:~$ sudo touch /etc/cassandra/service-enabled #T263617 |
[releng] |
| 21:40 |
<James_F> |
Docker: Pushing node12-test ano node12-test-browser 0.0.2 for T284492 |
[releng] |
| 21:35 |
<wm-bot> |
<lucaswerkmeister> deployed 547231388b (add create link for duplicates in bulk mode) |
[tools.lexeme-forms] |
| 21:26 |
<otto@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) |
[production] |
| 21:12 |
<sbassett> |
Deployed security patch for T284364 |
[production] |
| 20:04 |
<wm-bot> |
<lucaswerkmeister> deployed daf88503e0 (l10n updates) |
[tools.lexeme-forms] |
| 19:30 |
<ryankemper> |
T284479 [Cirrussearch] We'll keep monitoring. For now this incident is resolved. Glancing at our current volume relative to what we'd expect, the numbers we see match what we'd expect. If we're accidentally banning any innocent requests they must be an incredibly small percentage of the total otherwise we'd see significantly lower volume than expected |
[production] |
| 19:25 |
<ryankemper> |
T284479 [Cirrussearch] Seeing the expected drop in `entity_full_text` requests here: https://grafana-rw.wikimedia.org/d/000000455/elasticsearch-percentiles?viewPanel=47&orgId=1&from=now-12h&to=now As a result we're no longer rejecting any requests |
[production] |
| 19:21 |
<ryankemper> |
T284479 [Cirrussearch] We're working on rolling out https://gerrit.wikimedia.org/r/698607, which will ban search API requests that match the Google App Engine IP range `2600:1900::0/28` AND whose user agent includes `HeadlessChrome` |
[production] |
| 19:19 |
<cdanis> |
T284479 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕞🍵 sudo cumin -b16 'A:cp-text' "run-puppet-agent" |
[production] |
| 19:07 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@6199b67]: disable shelve/unshelve T284462 (duration: 04m 53s) |
[production] |
| 19:02 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@6199b67]: disable shelve/unshelve T284462 |
[production] |
| 19:01 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@6199b67]: disable shelve/unshelve (duration: 02m 01s) |
[production] |
| 18:59 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@6199b67]: disable shelve/unshelve |
[production] |
| 18:57 |
<herron> |
prometheus3001: moved /srv back to vda1 filesystem T243057 |
[production] |
| 18:39 |
<bstorm> |
cleaning up more error conditions on grid queues |
[tools] |
| 18:25 |
<urbanecm> |
[urbanecm@mwmaint1002 /srv/mediawiki/php-1.37.0-wmf.7]$ mwscript extensions/GrowthExperiments/maintenance/initWikiConfig.php --wiki=skwiki --phab=T284149 |
[production] |
| 18:24 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/GrowthExperiments/includes/WelcomeSurvey.php: 368b5d9: 0e79aee: WelcomeSurvey backports (T284127, T284257; 2/2) (duration: 00m 57s) |
[production] |
| 18:22 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/GrowthExperiments/extension.json: 368b5d9: 0e79aee: WelcomeSurvey backports (T284127, T284257; 1/2) (duration: 00m 56s) |
[production] |
| 18:20 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/GrowthExperiments/maintenance/initWikiConfig.php: 7089728: b2482fb: initWikiConfig GE backports (T284072) (duration: 00m 58s) |
[production] |
| 18:16 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 15e09109b7c45de967a496a0eb58ad267dbc5079: skwiki: Make Growth features available in dark mode (T284149; 3/3) (duration: 00m 56s) |
[production] |
| 18:14 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 15e09109b7c45de967a496a0eb58ad267dbc5079: skwiki: Make Growth features available in dark mode (T284149; 2/3) (duration: 00m 56s) |
[production] |
| 18:14 |
<otto@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers |
[production] |
| 18:14 |
<ottomata> |
rolling restart of kafka jumbo brokers - T283067 |
[production] |
| 18:14 |
<ottomata> |
rolling restart of kafka jumbo brokers - T283067 |
[analytics] |
| 18:13 |
<urbanecm@deploy1002> |
Synchronized wmf-config/config/skwiki.yaml: 15e09109b7c45de967a496a0eb58ad267dbc5079: skwiki: Make Growth features available in dark mode (T284149; 1/3) (duration: 00m 59s) |
[production] |
| 18:12 |
<otto@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) |
[production] |
| 18:04 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=skwiki growthexperiments # T284149 |
[production] |
| 18:04 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 5de2f8b27b016a2cd8f424d8e40318edde5e5704: Set WelcomeSurveyEnableWithHomepage (T281896, T284257) (duration: 00m 59s) |
[production] |
| 17:53 |
<otto@cumin1001> |
START - Cookbook sre.kafka.roll-restart-mirror-maker |
[production] |
| 17:53 |
<ottomata> |
rolling restart of kafka jumbo mirror makers - T283067 |
[production] |
| 17:53 |
<ottomata> |
rolling restart of kafka jumbo mirror makers - T283067 |
[analytics] |
| 17:42 |
<majavah> |
delete `ingress-nginx` namespace and related objects T264221 |
[tools] |
| 17:37 |
<majavah> |
remove tools-k8s-ingress-[1-3] from kubernetes, follow-up to https://sal.toolforge.org/log/nd7v2HkB1jz_IcWuCX5M T264221 |
[tools] |
| 17:17 |
<ryankemper> |
[Cirrussearch] We're seeing ~10% of current requests being rejected by poolcounter, due to ~2x expected `eqiad.full_text` query volume and ~30x expected `eqiad.entity_full_text` query volume |
[production] |
| 17:07 |
<ottomata> |
remove packages from an clsuter nodes: sudo apt-get -y remove r-cran-rmysql python3-matplotlib python3-sklearn python3-enchant python3-nltk gfortran liblapack-dev libopenblas-dev - T275786 |
[analytics] |
| 16:56 |
<ryankemper> |
[WDQS] `ryankemper@wdqs1005:~$ sudo systemctl restart wdqs-blazegraph` (blazegraph locked up) |
[production] |
| 16:51 |
<razzi> |
run homer '*.eqiad.wmnet' diff |
[production] |
| 16:50 |
<ottomata> |
restarting mysqld analytics-meta replica on db1108 to apply config change - T272973 |
[analytics] |
| 16:49 |
<ottomata> |
restarting mysqld analytics-meta replica on db1108 to apply config change - T272973 |
[production] |
| 16:31 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@19313f7]: Bump glent jar to 0.2.6 (duration: 04m 29s) |
[production] |
| 16:27 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@19313f7]: Bump glent jar to 0.2.6 |
[production] |
| 16:09 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@f236b95]: Bump glent jar to 0.2.6 (duration: 00m 35s) |
[production] |
| 16:09 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@f236b95]: Bump glent jar to 0.2.6 |
[production] |
| 14:57 |
<moritzm> |
installing remaining lz4 security updates on buster |
[production] |
| 14:35 |
<moritzm> |
installing isc-dhcp security updates |
[production] |
| 14:26 |
<andrewbogott> |
moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate T281399 |
[admin] |