951-1000 of 10000 results (23ms)
2020-06-22 §
11:53 <Urbanecm> EU B&C window done [production]
11:50 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.37/extensions/VisualEditor/modules/: Backport: 0a08066: Revert "Allow generic params to be passed to getWikitextFragment" (T255785) (duration: 00m 58s) [production]
11:45 <marostegui@cumin2001> dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P11627 and previous config saved to /var/cache/conftool/dbconfig/20200622-114554-marostegui.json [production]
11:40 <moritzm> draining ganeti2008 for eventual reboot [production]
11:37 <volans@deploy1001> Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 00m 28s) [production]
11:37 <volans@deploy1001> Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster [production]
11:34 <marostegui@cumin2001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11625 and previous config saved to /var/cache/conftool/dbconfig/20200622-113401-marostegui.json [production]
11:30 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 74e8295: IS: Cleanup some redundant rows (duration: 00m 56s) [production]
11:29 <Urbanecm> Run namespaceDupes.php for zh* projects (T165593) [production]
11:24 <marostegui@cumin2001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11623 and previous config saved to /var/cache/conftool/dbconfig/20200622-112451-marostegui.json [production]
11:24 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: db952ba: Add zh-hans and zh-hant translation of Module and Module_talk aliases for all Zh Projects (T165593) (duration: 00m 56s) [production]
11:16 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 1301fd4: Add import sources for gomwiktionary (T255098) (duration: 00m 57s) [production]
11:08 <marostegui@cumin2001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11622 and previous config saved to /var/cache/conftool/dbconfig/20200622-110806-marostegui.json [production]
11:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: defa81e: Disable NS_USER(_TALK) search engine indexing on trwiki (T255538) (duration: 00m 58s) [production]
10:35 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:606985| Bumping portals to master (606985)]] (duration: 00m 56s) [production]
10:34 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:606985| Bumping portals to master (606985)]] (duration: 01m 12s) [production]
09:58 <marostegui@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:56 <marostegui@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:33 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1094 for reimage', diff saved to https://phabricator.wikimedia.org/P11621 and previous config saved to /var/cache/conftool/dbconfig/20200622-093323-marostegui.json [production]
09:31 <godog> roll-restart logstash in codfw/eqiad to apply configuration change [production]
08:59 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:56 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:33 <moritzm> reimaging cumin1001 to buster T245114 [production]
08:13 <godog> extend prometheus codfw ops filesystem to 1TB [production]
08:02 <vgutierrez> upgrade to trafficserver 8.0.8~rc0-1wm1 on cp4026 and cp4032 [production]
08:02 <vgutierrez> upload trafficserver 8.0.8~rc0-1wm1 to apt.wm.o (buster) [production]
07:33 <marostegui@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:30 <marostegui@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:16 <marostegui> Reimage db1117 (irc haproxy alerts will be triggered) [production]
06:26 <marostegui@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:24 <marostegui@cumin2001> START - Cookbook sre.hosts.downtime [production]
06:06 <marostegui> Stop MySQL on dbstore1005 for reimage to Buster - T254870 [production]
05:58 <marostegui> Compress InnoDb on db1118 T254462 [production]
05:51 <marostegui@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:49 <marostegui@cumin2001> START - Cookbook sre.hosts.downtime [production]
05:43 <marostegui> Stop haproxy on dbproxy1008 - T255406 [production]
05:33 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1118 for reimage and InnoDB compression', diff saved to https://phabricator.wikimedia.org/P11617 and previous config saved to /var/cache/conftool/dbconfig/20200622-053334-marostegui.json [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1134', diff saved to https://phabricator.wikimedia.org/P11616 and previous config saved to /var/cache/conftool/dbconfig/20200622-053104-marostegui.json [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11615 and previous config saved to /var/cache/conftool/dbconfig/20200622-051730-marostegui.json [production]
05:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11614 and previous config saved to /var/cache/conftool/dbconfig/20200622-051720-marostegui.json [production]
05:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11613 and previous config saved to /var/cache/conftool/dbconfig/20200622-050259-marostegui.json [production]
04:50 <marostegui> Deploy schema change on s3 primary master with a big sleep between wikis - T250066 [production]
04:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P11612 and previous config saved to /var/cache/conftool/dbconfig/20200622-044853-marostegui.json [production]
2020-06-20 §
22:56 <cdanis@cumin2001> dbctl commit (dc=all): 'db1088 seems to have crashed', diff saved to https://phabricator.wikimedia.org/P11611 and previous config saved to /var/cache/conftool/dbconfig/20200620-225624-cdanis.json [production]
07:42 <elukey> powercycle an-worker1093 - bug soft lock up CPU showed in mgmt console [production]
07:36 <elukey> powercycle an-worker1091 - bug soft lock up CPU showed in mgmt console [production]
2020-06-19 §
18:10 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt - T238230 (duration: 00m 59s) [production]
16:07 <mutante> ganeti4003 - rebooting install4001 - trying to bootstrap OS install from install2003 [production]
15:47 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:28 <godog> roll-restart kibana to apply new settings [production]