601-650 of 10000 results (58ms)
2020-03-10 §
09:00 <marostegui> Start es5 deployment window T246072 [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1012', diff saved to https://phabricator.wikimedia.org/P10673 and previous config saved to /var/cache/conftool/dbconfig/20200310-085001-marostegui.json [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es1012 back to es1 master, this is a NOOP T239791', diff saved to https://phabricator.wikimedia.org/P10671 and previous config saved to /var/cache/conftool/dbconfig/20200310-082552-marostegui.json [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1012', diff saved to https://phabricator.wikimedia.org/P10670 and previous config saved to /var/cache/conftool/dbconfig/20200310-082525-marostegui.json [production]
05:36 <vgutierrez> restart ats-be on cp4032 to clean up the restart alert - T247232 [production]
2020-03-09 §
23:21 <catrope@deploy1001> Synchronized wmf-config/throttle.php: Remove expired throttle exemptions (duration: 01m 00s) [production]
23:15 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Create Define/Define talk: namespace on scowiki (duration: 01m 00s) [production]
16:20 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: revert: switch eventgate-analytics to use envoy (duration: 00m 59s) [production]
16:15 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch eventgate-analytics to use envoy (duration: 01m 05s) [production]
16:11 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
15:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1012', diff saved to https://phabricator.wikimedia.org/P10668 and previous config saved to /var/cache/conftool/dbconfig/20200309-154627-marostegui.json [production]
15:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1012', diff saved to https://phabricator.wikimedia.org/P10667 and previous config saved to /var/cache/conftool/dbconfig/20200309-153515-marostegui.json [production]
15:29 <marostegui> Upgrade mysql on es1012 T239791 [production]
15:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2125 - T246604', diff saved to https://phabricator.wikimedia.org/P10666 and previous config saved to /var/cache/conftool/dbconfig/20200309-152427-marostegui.json [production]
15:18 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch mathoid to use envoy (duration: 00m 59s) [production]
15:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1012 T239791', diff saved to https://phabricator.wikimedia.org/P10665 and previous config saved to /var/cache/conftool/dbconfig/20200309-151751-marostegui.json [production]
15:14 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
15:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote es1016 to es1 master, this is a NOOP T239791', diff saved to https://phabricator.wikimedia.org/P10664 and previous config saved to /var/cache/conftool/dbconfig/20200309-151310-marostegui.json [production]
15:13 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
15:12 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
15:06 <marostegui> Restart mysql on db1116 (the previous one was db1102) for upgrade [production]
14:57 <marostegui> Restart mysql for upgrade [production]
14:56 <hoo> Updated the Wikidata property suggester with data from the 2020-03-02 JSON dump and applied the T132839 workarounds [production]
14:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1121 T239791', diff saved to https://phabricator.wikimedia.org/P10663 and previous config saved to /var/cache/conftool/dbconfig/20200309-145232-marostegui.json [production]
14:48 <marostegui> Restart and upgrade mysql on db1121 T239791 [production]
14:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 T239791', diff saved to https://phabricator.wikimedia.org/P10662 and previous config saved to /var/cache/conftool/dbconfig/20200309-144752-marostegui.json [production]
14:41 <godog> roll restart logstash in codfw / eqiad - T226986 [production]
13:52 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . [production]
13:49 <oblivian@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' . [production]
12:30 <akosiaris> upload apertium 3.6.1, cg3 1.3.1, lttoolbox 3.5.1, apertium-lex-tools 0.2.3 to apt.wikimedia.org/jessie-wikimedia main. T234182 [production]
12:06 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch sessionstore to use envoy permanently (duration: 00m 59s) [production]
11:25 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: test: switch sessionstore to use envoy again (duration: 00m 57s) [production]
11:10 <Amir1> EU SWAT is done [production]
11:09 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577671|Add `fkv` Kven to $wmgExtraLanguageNames (T167259)]], take II (duration: 00m 57s) [production]
11:08 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577671|Add `fkv` Kven to $wmgExtraLanguageNames (T167259)]] (duration: 00m 59s) [production]
10:58 <vgutierrez> upload pystemd 0.7.0-1wm1 to apt.wm.o (buster) - T245616 [production]
10:46 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:578305| Bumping portals to master (563985)]] (duration: 00m 58s) [production]
10:45 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:578305| Bumping portals to master (563985)]] (duration: 00m 58s) [production]
10:34 <moritzm> install spamassassin security updates on fermium/lists.wikimedia.org [production]
10:32 <moritzm> install spamassassin security updates on mendelevium/ticket.wikimedia.org [production]
10:26 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . [production]
10:12 <moritzm> installing openjdk-7 security updates [production]
10:04 <vgutierrez> disable parent proxies globally on ats-tls - T244464 [production]
10:00 <moritzm> installing php5 security updates [production]
09:51 <gehel> pooling new elastic20[55-60] servers - T246975 [production]
09:48 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: re-revert: switch sessionstore to use envoy (duration: 00m 35s) [production]
09:39 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: re-try: switch sessionstore to use envoy (duration: 00m 58s) [production]
09:14 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: revert switch sessionstore to use envoy (duration: 00m 58s) [production]
09:08 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch sessionstore to use envoy (duration: 01m 00s) [production]
09:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]