|
2018-07-01
§
|
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:14 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 03:13 |
<hjhjjki> |
forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 v |
[production] |
| 02:54 |
<hjukhjuki> |
forcing reboot of cp3033 via mgmt |
[production] |
| 02:53 |
<hjukhjuki> |
forcing reboot of labservices1001 via mgmt |
[production] |
| 02:44 |
<andrewbogott> |
forcing reboot of labservices1001 via mgmt |
[production] |
|
2018-06-30
§
|
| 21:35 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s) |
[production] |
| 21:32 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 |
[production] |
| 20:40 |
<Krenair> |
ran git gc on deployment-tin:/srv/mediawiki to free up space |
[releng] |
| 20:40 |
<Krenair> |
ran git gc on deployment-tin:/srv/mediawiki to free up space |
[deployment-prep] |
| 18:15 |
<chicocvenancio> |
pushed new config to PAWS to fix dumps nfs mountpoint |
[tools] |
| 16:40 |
<zhuyifei1999_> |
because tools-paws-master-01 was having ~1000 loadavg due to NFS having issues and processes stuck in D state |
[tools] |
| 16:39 |
<zhuyifei1999_> |
reboot tools-paws-master-01 |
[tools] |
| 16:35 |
<zhuyifei1999_> |
`root@tools-paws-master-01:~# sed -i 's/^labstore1006.wikimedia.org/#labstore1006.wikimedia.org/' /etc/fstab` |
[tools] |
| 16:34 |
<andrewbogott> |
"sed -i '/labstore1006/d' /etc/fstab" everywhere |
[tools] |
| 16:28 |
<chasemp> |
labstore1006:~# service nfs-kernel-server restart |
[production] |
|
2018-06-29
§
|
| 23:49 |
<bd808> |
Stopped webservice due to CrashLoopBackOff failures. Caused by "ImportError: No module named 'yamlconf'" |
[tools.ores-support-checklist] |
| 23:44 |
<bd808> |
test |
[tools.stashbot] |
| 21:37 |
<chasemp> |
(late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad |
[production] |
| 19:11 |
<herron> |
stat1005:~# killall -u ironholds T197895 |
[production] |
| 17:41 |
<bd808> |
Rescheduling continuous jobs away from tools-exec-1408 where load is high |
[tools] |
| 17:28 |
<XioNoX> |
Re-activating v6 BGP session to Telia on cr2-eqiad - T198502 |
[production] |
| 17:21 |
<XioNoX> |
deactivating v6 BGP session to Telia on cr2-eqiad - T198502 |
[production] |
| 17:11 |
<bd808> |
Rescheduled jobs away from toole-exec-1404 where linkwatcher is currently stealing most of the CPU (T123121) |
[tools] |
| 16:46 |
<bd808> |
Killed orphan tool owned processes running on the job grid. Mostly jembot and wsexport php-cgi processes stuck in deadlock following an OOM. T182070 |
[tools] |
| 14:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 00m 50s) |
[production] |
| 12:30 |
<moritzm> |
uploaded ffmpeg 3.2.10-1~deb9u1+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (linked against libvpx 1.7 and with backported row-mt support) (T190333) |
[production] |
| 11:11 |
<moritzm> |
installing zsh updates from jessie 8.11 point release |
[production] |
| 11:01 |
<moritzm> |
installing lame security updates |
[production] |
| 10:34 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2058 after reimage (duration: 00m 51s) |
[production] |
| 10:21 |
<mobrovac@deploy1001> |
Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 (duration: 03m 26s) |
[production] |
| 10:20 |
<marostegui> |
Deploy schema change on db1092 T191316 T192926 T89737 T195193 |
[production] |
| 10:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 00m 50s) |
[production] |
| 10:18 |
<mobrovac@deploy1001> |
Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 |
[production] |
| 10:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 00m 50s) |
[production] |