2401-2450 of 10000 results (73ms)
2020-02-24 §
13:01 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573965|Disallow crats to (un)assign flow-bot group on enwiki (T245716)]] (duration: 00m 56s) [production]
12:59 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573965|Disallow crats to (un)assign flow-bot group on enwiki (T245716)]] (duration: 00m 56s) [production]
12:48 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:574398| Bumping portals to master (563985)]] (duration: 00m 56s) [production]
12:47 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:574398| Bumping portals to master (563985)]] (duration: 00m 56s) [production]
12:38 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:571738|Add definitions for redirect badges (T235420)]], take II, the cache issue (duration: 00m 56s) [production]
12:37 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:571738|Add definitions for redirect badges (T235420)]] (duration: 00m 56s) [production]
12:23 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/Wikibase/client/includes: SWAT: [[gerrit:574391|Use formatter cache in client LUA label lookups (T245740)]] (duration: 00m 56s) [production]
12:19 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/WikimediaMaintenance/dumpInterwiki.php: dumpInterwiki: Respect comments in dblists (T244906) (duration: 00m 56s) [production]
12:12 <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|574265|CX: Adjust MT threshold for Telugu WP to 70% (T244769)]] (duration: 00m 56s) [production]
12:05 <XioNoX> re-enable deactivated BGP sessions from ulsfo to office - T239893 [production]
12:02 <vgutierrez> reimage pybal-test2001 as buster - T224570 T245984 [production]
11:49 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:574398| Bumping portals to master (563985)]] (duration: 00m 55s) [production]
11:45 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:574398| Bumping portals to master (563985)]] (duration: 00m 57s) [production]
11:27 <vgutierrez> upload pybal 1.15.8 to apt.wm.o (buster) - T245984 [production]
11:06 <volans> restarted ferm on ms-be2046 [production]
11:02 <marostegui> Move labsdb1009, labsdb1011 and labsdb1012 (labsdb1010 is currently delayed, will be done later) to replicate under codfw for a few days while we alter wb_terms on db1087 - T232446 [production]
10:59 <effie> upgrading scap in eqiad and codfw - T245530 [production]
10:55 <volans> restarted ferm on ms-be2016, had failed with DNS query for 'ms-be2056.codfw.wmnet' failed: query timed out [production]
10:41 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/Wikibase: [[gerrit:574386|Add metric for recording cache hits in StatsdRecordingSimpleCache]] (T244260) (duration: 01m 04s) [production]
10:34 <godog> onboard netbox to logging pipeline [production]
10:12 <marostegui> Stop db1087 and db2079 in sync - T232446 [production]
10:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 for compression and place db1101:3318 into vslow,dump - T232446', diff saved to https://phabricator.wikimedia.org/P10493 and previous config saved to /var/cache/conftool/dbconfig/20200224-101030-marostegui.json [production]
09:21 <godog> bounce ferm on ms-be2023, it had failed (no entries in journald) [production]
09:08 <elukey> update puppet compiler's facts [production]
08:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es5 eqiad - T245806', diff saved to https://phabricator.wikimedia.org/P10492 and previous config saved to /var/cache/conftool/dbconfig/20200224-084027-marostegui.json [production]
08:33 <marostegui@deploy1001> Synchronized wmf-config/etcd.php: Add es4 and es5 (unused new external store sections to etcd - T245806 (duration: 00m 58s) [production]
08:29 <marostegui> Temporary put es1020 (es4) and es1023 (es5) on RO on a mysql level - T245806 [production]
08:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es5 codfw - T245806', diff saved to https://phabricator.wikimedia.org/P10491 and previous config saved to /var/cache/conftool/dbconfig/20200224-082848-marostegui.json [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es4 eqiad - T245806', diff saved to https://phabricator.wikimedia.org/P10490 and previous config saved to /var/cache/conftool/dbconfig/20200224-080708-marostegui.json [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es4 codfw - T245806', diff saved to https://phabricator.wikimedia.org/P10489 and previous config saved to /var/cache/conftool/dbconfig/20200224-080128-marostegui.json [production]
07:31 <cdanis> dbctl: edit es4/es5 sections in eqiad (flavor & master & min_replicas fields) T245806 [production]
07:30 <cdanis> dbctl: (and min_replicas field) T245806 [production]
07:29 <cdanis> dbctl: edit es4/es5 sections in codfw (flavor & master fields) T245806 [production]
07:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1107 for 10.4 testing in special slaves group with weight 10 - T242702', diff saved to https://phabricator.wikimedia.org/P10488 and previous config saved to /var/cache/conftool/dbconfig/20200224-071201-marostegui.json [production]
07:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1107 for 10.4 testing in main and API - T242702', diff saved to https://phabricator.wikimedia.org/P10487 and previous config saved to /var/cache/conftool/dbconfig/20200224-070337-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1101:3318 after removing partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10486 and previous config saved to /var/cache/conftool/dbconfig/20200224-064044-marostegui.json [production]
06:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318 after removing partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10485 and previous config saved to /var/cache/conftool/dbconfig/20200224-063258-marostegui.json [production]
06:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318 after removing partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10484 and previous config saved to /var/cache/conftool/dbconfig/20200224-062226-marostegui.json [production]
06:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318 after removing partitions - T239453', diff saved to https://phabricator.wikimedia.org/P10483 and previous config saved to /var/cache/conftool/dbconfig/20200224-060118-marostegui.json [production]
05:57 <marostegui> Repool labsdb1011 - T245797 [production]
2020-02-23 §
16:52 <elukey> powercycle mw1372 - no mgmt console, no ssh [production]
15:17 <Urbanecm> mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='𐰇𐱅𐰚𐰤' /home/urbanecm/T245950 (T245950) [production]
2020-02-22 §
03:41 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
03:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
02:17 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
02:16 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
02:13 <mutante> ganeti - removing instances apt1001/apt2001 again, starting over [production]
01:53 <mutante> starting new ganeti VMs apt1001 and apt2001 for OS install (WIP, not prod) [production]
01:03 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
01:01 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]