701-750 of 10000 results (15ms)
2020-10-05 §
15:15 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:56 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:55 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:41 <elukey> shutdown stat1005 and stat1008 for ram expansion (1005 again) [production]
14:36 <ppchelko@deploy1001> Finished deploy [restbase/deploy@366a543]: T263133 T264035 (duration: 22m 23s) [production]
14:25 <elukey> shutdown an-master1001 for ram expansion [production]
14:13 <ppchelko@deploy1001> Started deploy [restbase/deploy@366a543]: T263133 T264035 [production]
14:01 <filippo@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
13:58 <filippo@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
13:55 <filippo@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
13:54 <elukey> shutdown stat1005 for ram upgrade [production]
13:31 <elukey> shutdown an-master1002 for ram expansion (64 -> 128G) [production]
12:39 <moritzm> installing curl security updates on remaining hosts [production]
11:34 <hoo@deploy1001> Synchronized wmf-config/: Revert "Remove $wgExtraLanguageNames from Wikidata and Commons" (T264295) (duration: 00m 59s) [production]
11:28 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: be73f155001e9095697c3c21a208c63e7bf5d2d1: Move changetags right from users to sysop [trwiki] (T264508) (duration: 00m 59s) [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cd30b626e23b48146b970c72731f8f7bb1eee9e1: wgSkipSkins: Exclude contenttranslation skin from skin options for users (T263093) (duration: 00m 59s) [production]
11:05 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:632212| Bumping portals to master (T128546)]] (duration: 00m 58s) [production]
11:04 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:632212| Bumping portals to master (T128546)]] (duration: 00m 58s) [production]
10:37 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
10:37 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:632204| Bumping portals to master (T128546)]] (duration: 00m 58s) [production]
10:36 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:632204| Bumping portals to master (T128546)]] (duration: 01m 00s) [production]
10:34 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
10:32 <ema> cp3052: pool with varnish 5.1.3-1wm15 T264398 [production]
10:28 <ema> cp3052: depool and downgrade varnish to 5.1.3-1wm15 T264398 [production]
10:08 <moritzm> installing ldap-replica1002 T264390 [production]
09:52 <moritzm> installing ldap-replica1001 T264390 [production]
09:22 <moritzm> installing ldap-replica2003 T264390 [production]
09:02 <hnowlan> bootstrapping restbase1030-b [production]
08:57 <moritzm> installing ldap-replica2004 T264390 [production]
08:40 <kormat@cumin1001> dbctl commit (dc=all): 'db2073 depooling: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12918 and previous config saved to /var/cache/conftool/dbconfig/20201005-084022-kormat.json [production]
08:39 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:39 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:38 <kormat@cumin1001> dbctl commit (dc=all): 'Add db2119 to s4 dump/vslow temporarily T259831', diff saved to https://phabricator.wikimedia.org/P12917 and previous config saved to /var/cache/conftool/dbconfig/20201005-083822-kormat.json [production]
08:23 <godog> prometheus codfw/ops, add 100G to the LV [production]
08:06 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
07:46 <marostegui> Stop mysql on es2017 T264386 [production]
07:30 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
06:52 <XioNoX> add static NAT to pfw3-eqiad - T264356 [production]
06:33 <elukey> reboot stat1005 to resolve weird GPU state (scheduled last week) [production]
05:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2017 T264386 ', diff saved to https://phabricator.wikimedia.org/P12916 and previous config saved to /var/cache/conftool/dbconfig/20201005-050636-marostegui.json [production]
2020-10-03 §
15:52 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: emergency: 840545f1d9115ea6b672cecce1762d850d8b1f54: Restrict flow-hide right to autoconfirmed users on zhwiki (T264489) (duration: 01m 17s) [production]
00:08 <ejegg> updated fundraising CiviCRM from 256adda03c to a30da7f92a [production]
2020-10-02 §
22:00 <mutante> depooling mw2271 because Icinga alerts about memcached and SAL shows there were ongoing tests of some kind on it [production]
21:59 <dzahn@cumin1001> conftool action : set/pooled=no; selector: dc=codfw,name=mw2271.codfw.wmnet [production]
21:35 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:32 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
21:26 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:22 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
19:14 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:35 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]