4501-4550 of 10000 results (95ms)
2018-07-02 §
04:59 <marostegui> Deploy schema change on db1109 T191316 T192926 T89737 T195193 [production]
04:58 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 00m 56s) [production]
04:25 <twentyafterfour> Bsadowski1 The throttle is averaged over 5 minutes, you should be unblocked shortly after being blocked [production]
04:24 <twentyafterfour> applied security patch on phab1001, source in /srv/patches [production]
02:44 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Mon Jul 2 02:44:40 UTC 2018 (duration 10m 21s) [production]
02:34 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 39s) [production]
00:25 <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023 [releng]
2018-07-01 §
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:14 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt [production]
03:13 <hjhjjki> forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 v [production]
02:54 <hjukhjuki> forcing reboot of cp3033 via mgmt [production]
02:53 <hjukhjuki> forcing reboot of labservices1001 via mgmt [production]
02:44 <andrewbogott> forcing reboot of labservices1001 via mgmt [production]
2018-06-30 §
21:35 <ariel@deploy1001> Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s) [production]
21:32 <ariel@deploy1001> Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 [production]
20:40 <Krenair> ran git gc on deployment-tin:/srv/mediawiki to free up space [releng]
20:40 <Krenair> ran git gc on deployment-tin:/srv/mediawiki to free up space [deployment-prep]
18:15 <chicocvenancio> pushed new config to PAWS to fix dumps nfs mountpoint [tools]
16:40 <zhuyifei1999_> because tools-paws-master-01 was having ~1000 loadavg due to NFS having issues and processes stuck in D state [tools]
16:39 <zhuyifei1999_> reboot tools-paws-master-01 [tools]
16:35 <zhuyifei1999_> `root@tools-paws-master-01:~# sed -i 's/^labstore1006.wikimedia.org/#labstore1006.wikimedia.org/' /etc/fstab` [tools]
16:34 <andrewbogott> "sed -i '/labstore1006/d' /etc/fstab" everywhere [tools]
16:28 <chasemp> labstore1006:~# service nfs-kernel-server restart [production]
2018-06-29 §
23:49 <bd808> Stopped webservice due to CrashLoopBackOff failures. Caused by "ImportError: No module named 'yamlconf'" [tools.ores-support-checklist]
23:44 <bd808> test [tools.stashbot]
21:37 <chasemp> (late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad [production]
19:11 <herron> stat1005:~# killall -u ironholds T197895 [production]
17:41 <bd808> Rescheduling continuous jobs away from tools-exec-1408 where load is high [tools]
17:28 <XioNoX> Re-activating v6 BGP session to Telia on cr2-eqiad - T198502 [production]
17:21 <XioNoX> deactivating v6 BGP session to Telia on cr2-eqiad - T198502 [production]
17:11 <bd808> Rescheduled jobs away from toole-exec-1404 where linkwatcher is currently stealing most of the CPU (T123121) [tools]
16:46 <bd808> Killed orphan tool owned processes running on the job grid. Mostly jembot and wsexport php-cgi processes stuck in deadlock following an OOM. T182070 [tools]