8401-8450 of 10000 results (60ms)
2017-06-30 §
05:44 <marostegui> Deploy alter table on dbstore1002 on s2 - T168661 [production]
05:43 <marostegui> Deploy alter table on dbstore1001 on s2 - T168661 [production]
05:37 <marostegui> Deploy alter table on db1069 (and let it replicate) on s2 - T168661 [production]
02:29 <chasemp> labstore1005 start drbd [production]
02:14 <chasemp> reboot labstore1005 (5m ago) [production]
01:25 <chasemp> reboot labstoer1005 [production]
01:23 <chasemp> fail nfs from labstore1005 to labstore1004 (I failed to log a previous failover to 1004 and back) [production]
2017-06-29 §
23:32 <RoanKattouw> Sorry I meant T169163 [production]
23:31 <catrope@tin> Synchronized php-1.30.0-wmf.7/resources/src/mediawiki.rcfilters/: RCFilters fixes (T169169, T169107, T169042) (duration: 00m 42s) [production]
23:28 <catrope@tin> Synchronized php-1.30.0-wmf.7/extensions/WikimediaEvents/: Add event logging for explode-similar on SRP (T149809) (duration: 00m 42s) [production]
23:26 <catrope@tin> Synchronized php-1.30.0-wmf.7/extensions/CirrusSearch/: "Explore similar" widget for CirrusSearch (T149809) (duration: 00m 54s) [production]
23:19 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Stop reader surveys (T131949) (duration: 00m 43s) [production]
22:16 <chasemp> set cfq scheduler on labstore1005 [production]
21:51 <mutante> APT - uploading python-django-south from jessie to wikimedia-stretch for librenms on stretch (T159756) [production]
21:40 <chasemp> reboot labstore1004 with grub set to gnulinux-advanced-1773f282-5a1b-441e-865c-8b70a0ebc925>gnulinux-4.4.0-3-amd64-advanced-1773f282-5a1b-441e-865c-8b70a0ebc925 [production]
21:10 <mobrovac@tin> Finished deploy [restbase/deploy@bcb83f4]: (no justification provided) (duration: 04m 21s) [production]
21:06 <mobrovac@tin> Started deploy [restbase/deploy@bcb83f4]: (no justification provided) [production]
21:06 <mobrovac@tin> Finished deploy [restbase/deploy@bcb83f4]: (no justification provided) (duration: 01m 02s) [production]
21:05 <mobrovac@tin> Started deploy [restbase/deploy@bcb83f4]: (no justification provided) [production]
21:04 <mobrovac@tin> Finished deploy [restbase/deploy@bcb83f4]: Fix special char handling in PDF back-end requests - T169223 (duration: 03m 14s) [production]
21:00 <mobrovac@tin> Started deploy [restbase/deploy@bcb83f4]: Fix special char handling in PDF back-end requests - T169223 [production]
20:46 <mutante> APT - reprepro copy stretch-wikimedia jessie-wikimedia prometheus-snmp-exporter (to make it available on stretch for netmon1002) (T159756) [production]
20:39 <ppchelko@tin> Finished deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 (duration: 01m 01s) [production]
20:38 <ppchelko@tin> Started deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 [production]
19:23 <demon@tin> Synchronized scap/plugins/clean.py: Because I need to learn basic python syntax before trying stuff (duration: 00m 42s) [production]
19:21 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.7 refs T167536 [production]
19:16 <twentyafterfour> deploying wmf/1.30.0-wmf.7 to all wikis refs T167536 [production]
19:06 <demon@tin> Pruned MediaWiki: 1.30.0-wmf.5 [keeping static files] (duration: 01m 16s) [production]
18:30 <chasemp> restart nfs on labstore1004 (primary) [production]
18:10 <demon@tin> Synchronized php-1.30.0-wmf.7/extensions/TextExtracts/extension.json: T107206 (duration: 00m 47s) [production]
17:28 <arlolra> Updated Parsoid to b4187f18 (T168900, T168675, T168404, T153203) [production]
17:21 <arlolra@tin> Finished deploy [parsoid/deploy@717df08]: Updating Parsoid to b4187f18 (duration: 09m 41s) [production]
17:12 <arlolra@tin> Started deploy [parsoid/deploy@717df08]: Updating Parsoid to b4187f18 [production]
17:08 <mobrovac> scb2005 repooling back the services - T167763 [production]
16:21 <godog> temporarily stop ircecho, puppet spam [production]
16:05 <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 46s) [production]
15:40 <akosiaris> disable puppet on all of eqiad/esams, problems with ganeti and puppetdb [production]
15:38 <chasemp> restart nfs-exportd on labstore1004 [production]
15:34 <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 02m 54s) [production]
15:26 <mobrovac> scb2005 depooled all services for T167763 [production]
15:09 <chasemp> set downtimes for labstore1004/1005 failover see https://etherpad.wikimedia.org/p/labstore_reboots [production]
15:02 <akosiaris> purge d-i-test from puppet/salt [production]
14:57 <akosiaris> reboot aluminium.wikimedia.org bromine.eqiad.wmnet etherpad1001.eqiad.wmnet d-i-test.eqiad.wmnet kubestagetcd1001.eqiad.wmnet mx1001.wikimedia.org seaborgium.wikimedia.org for kernel upgrades [production]
14:47 <jynus> several restarts of db2072 services and host on the following hour [production]
14:30 <ema> varnish 4.1.7-1wm1 uploaded to apt.w.o, cp1008 upgraded T164768 [production]
14:08 <marostegui> Deploy alter table on s7 on dbstore1001 - T166208 [production]
13:54 <godog> kick sdb out of mdadm arrays on bast3002 - T169035 [production]
12:56 <akosiaris@tin> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 46s) [production]
12:47 <akosiaris> reboot argon.eqiad.wmnet, darmstadtium.eqiad.wmnet, dbmonitor1001.wikimedia.org, etcd1001.eqiad.wmnet, etcd1006.eqiad.wmnet, krypton.eqiad.wmnet, mendelevium.eqiad.wmnet, mwdebug1001.eqiad.wmnet, roentgenium.eqiad.wmnet, sca1003.eqiad.wmnet for kernel upgrades [production]
12:41 <akosiaris> reboot poolcounter1001 for kernel upgrades [production]