1-50 of 10000 results (27ms)
2019-01-26 §
11:06 <volans> force rebooting icinga1001 (no ping, no ssh, stuck console) [production]
03:23 <marostegui> Convert all tables on incubatorwiki to innodb to fix s3 thread - T213670 [production]
00:03 <XioNoX> split member-range ge-3/0/0 to ge-3/0/38 on asw-b-codfw [production]
2019-01-25 §
22:45 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@5e859c4]: Update mobileapps to a8834e8 (T214728) (duration: 03m 27s) [production]
22:42 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@5e859c4]: Update mobileapps to a8834e8 (T214728) [production]
21:56 <krinkle@deploy1001> Synchronized wmf-config/flaggedrevs.php: I95c37d628557c (duration: 00m 46s) [production]
21:44 <krinkle@deploy1001> Synchronized wmf-config/: Idb695dd033d42 (duration: 00m 46s) [production]
21:43 <krinkle@deploy1001> Synchronized wmf-config/PhpAutoPrepend.php: Idb695dd033d42 (duration: 00m 47s) [production]
21:05 <robh> cleared sel on db1068, it had a power redundancy loss event (old and resolved) that was triggering the icinga check [production]
20:04 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1106 as an extra api host (duration: 00m 46s) [production]
19:36 <jynus> powercycle db1114 T214720 [production]
19:21 <jynus> disabling notifications on db1114 [production]
19:21 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 00m 46s) [production]
18:32 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@94b76f5]: Update mobileapps to 4c42e3d (T214714) (duration: 03m 33s) [production]
18:28 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@94b76f5]: Update mobileapps to 4c42e3d (T214714) [production]
17:17 <chaomodus> notebook1003 restarted nagios-nrpe-server due to oom - T212824 [production]
14:43 <hashar> contint1001: stopping zuul-merger for cleanup duties [production]
09:48 <marostegui> Add dbstore1005:3318 to tendril T210478 [production]
08:24 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1105 (duration: 00m 45s) [production]
08:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Give more traffic to db1105:3312 (duration: 00m 45s) [production]
07:51 <elukey> restart yarn/hdfs daemons on analytics1056 to pick up new disk settings - T214057 [production]
07:40 <elukey> drain + reboot analytics1054 after disk swap (verify reboot + restore correct fstab mountpoints) - T213038 [production]
07:30 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1105:3312 (duration: 00m 45s) [production]
07:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1105 (duration: 00m 47s) [production]
06:53 <marostegui> Stop MySQL on db1105 to upgrade MySQL [production]
06:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully depool db1105 (duration: 00m 46s) [production]
06:49 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1122 T210713 (duration: 00m 47s) [production]
06:13 <marostegui> Deploy schema change on db1122 - T210713 [production]
06:12 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1122 T210713 (duration: 00m 48s) [production]
06:04 <marostegui> Compress dbstore1002: staging.mep_word_persistence from Aria to InnoDB - T213706 [production]
05:42 <kartik@deploy1001> Finished deploy [cxserver/deploy@a5d7181]: Update cxserver to 356f0a1 (T213257, T213275) (duration: 04m 09s) [production]
05:38 <kartik@deploy1001> Started deploy [cxserver/deploy@a5d7181]: Update cxserver to 356f0a1 (T213257, T213275) [production]
03:12 <mutante> scandium sudo chgrp -R wikidev /srv/deployment/parsoid/deploy/ ; sudo chmod -R g+w /srv/deployment/parsoid/deploy/ (T201366) [production]
03:03 <mutante> scandium - apt-get -t stretch-backports install npm ; run puppet ; remove manually created /apt/preferences.d/npm.pref ; puppet created npm_stretch_backports.pref ; puppet run without errors again (T201366) [production]
01:33 <crusnov@deploy1001> Finished deploy [netbox/deploy@7770453]: Cleanup deploy - T212524 (duration: 00m 11s) [production]
01:33 <crusnov@deploy1001> Started deploy [netbox/deploy@7770453]: Cleanup deploy - T212524 [production]
01:28 <crusnov@deploy1001> Finished deploy [netbox/deploy@7770453]: Upgrade netbox to 2.5.3 - T212524 Try 2 (duration: 00m 31s) [production]
01:27 <crusnov@deploy1001> Started deploy [netbox/deploy@7770453]: Upgrade netbox to 2.5.3 - T212524 Try 2 [production]
01:26 <crusnov@deploy1001> Finished deploy [netbox/deploy@7770453]: Upgrade netbox to 2.5.3 - T212524 (duration: 07m 43s) [production]
01:18 <crusnov@deploy1001> Started deploy [netbox/deploy@7770453]: Upgrade netbox to 2.5.3 - T212524 [production]
00:46 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT T214515 gerrit:486154: Turn on wbsearchentities ab test in de, fr, es (duration: 00m 46s) [production]
00:37 <ebernhardson@deploy1001> Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T214515 gerrit:484334: Add wbsearchentities profiles for de, fr, es (duration: 00m 45s) [production]
00:34 <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.14/extensions/MobileFrontend/: SWAT T214606 gerrit:486392: MobileFrontend if wikidatadata description exists, set it as tagline (duration: 00m 47s) [production]
00:29 <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.14/includes/Title.php: SWAT T210739 gerrit:486369: Clone the Title object to prevent mutation (duration: 00m 47s) [production]
00:20 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: SWAT T212788 gerrit:485609: autocomplete subphrase matching on wikitech and mw.org 2 of 2 (duration: 00m 45s) [production]
00:14 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT T212788 gerrit:485608: autocomplete subphrase matching on wikitech and mw.org (duration: 00m 46s) [production]
00:01 <arlolra> Updated Parsoid to 4772f44 (T214649, T214648) [production]
2019-01-24 §
23:54 <arlolra@deploy1001> Finished deploy [parsoid/deploy@f9ef630]: Updating Parsoid to 4772f44 (duration: 11m 58s) [production]
23:42 <arlolra@deploy1001> Started deploy [parsoid/deploy@f9ef630]: Updating Parsoid to 4772f44 [production]
22:21 <mutante> wikitech-static splitting apache2 config files into one file per vhost to make it possible for certbot t odetect them [production]