Ubuntu14.04关闭错误报告

错误报告总是弹出来,不知道什么问题,又无法解决问题,不如直接关掉。

Ubuntu14.04关闭错误报告的方法:

>> sudo emacs /etc/default/apport
enabled=0
>> sudo reboot

KVM克隆虚拟机

1、启动virt-manager,右击cn001,点击"Clone...",在弹出窗口的Name中输入"cn002",点击"Clone"。

2、克隆完成后,启动cn002。

3、修改hostname

>> sudo emacs /etc/hostname
cn002

4、修改IP地址

>> sudo emacs /etc/network/interfaces
address 192.168.1.212

5、重启就可以正常使用cn002了。

使用pexpect远程登录SSH

pexpect可以自动和命令行程序进行交互,在系统部署和测试的时候非常好用。

以远程登录SSH并执行命令为例

>> emacs dparkstarter.py
import pexpect

ip = "192.168.1.211"
name = "long"
child = pexpect.spawn('ssh  %s@%s' % (name, ip))
child.expect('$')
child.sendline('python dparkslave.py')
child.interact()
>> python dparkstarter.py

程序可以自动登录SSH,然后执行python dparkslave.py这条命令。

Ubuntu14.04部署分布式计算框架Dpark(3)

在cn001上

1、安装ssh和emacs

>> sudo apt-get update
>> sudo apt-get install ssh openssh-server openssh-client emacs

2、设定静态IP,参考文章 https://codinglonglong.github.io/posts/ubuntu1404bu-shu-tornado%2Bnginx%2Bsupervisor/

3、重启系统

4、安装Beansdb的Python接口

>> cd ~
>> sudo apt-get install build-essential gcc g++ gfortran git python-pip python-dev autoconf automake libtool
>> scp long@192.168.1.203:/home/long/libmemcached-1.0.18.tar.gz .
>> tar zxvf libmemcached-1.0.18.tar.gz
>> cd libmemcached-1.0.18/
>> ./configure
>> sudo make
>> sudo make install
>> git clone https://github.com/douban/python-libmemcached
>> cd python-libmemcached/
>> sudo python setup.py install
>> emacs ~/.profile
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
>> source ~/.profile
>> sudo scp long@192.168.1.203:/home/long/beansdb/python/dbclient.py /usr/local/lib/python2.7/dist-packages/

5、安装MooseFS的客户端

>> cd ~
>> scp long@192.168.1.203:/home/long/fuse-2.9.4.tar.gz .
>> scp long@192.168.1.203:/home/long/zlib-1.2.8.tar.gz .
>> scp long@192.168.1.203:/home/long/moosefs-2.0.73-1.tar.gz .
>> tar zxvf fuse-2.9.4.tar.gz && cd fuse-2.9.4 && ./configure --prefix=/usr && sudo make && sudo make install && cd ..
>> tar zxvf zlib-1.2.8.tar.gz && cd zlib-1.2.8 && ./configure && sudo make && sudo make install && cd ..
>> sudo useradd mfs -M -s /sbin/nologin
>> sudo apt-get install libpcap-dev pkg-config
>> tar zxvf moosefs-2.0.73-1.tar.gz && cd moosefs-2.0.73 && ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --enable-mfsmount && sudo make && sudo make install && cd ..
>> cd /usr/local/mfs/etc/mfs
>> sudo cp mfsmount.cfg.dist mfsmount.cfg
>> sudo emacs mfsmount.cfg
# Example:
#
/mnt/mfs
>> sudo mkdir /mnt/mfs
>> sudo chown -R mfs:mfs /mnt/mfs

6、建立自启动脚本

>> cd /usr/local/bin
>> sudo emacs startmfsmount
cd /usr/local/mfs/bin/
./mfsmount -H 192.168.1.203 -o nonempty
>> sudo chmod 777 startmfsmount

7、设定无密码启动

>> sudo visudo
long ALL=NOPASSWD:/usr/local/bin/startmfsmount

8、安装Mesos的客户端

>> scp -r long@192.168.1.203:/home/long/mesos /home/long
>> sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev autoconf libtool

9、建立自启动脚本

>> cd /usr/local/bin
>> sudo emacs startmesosslave
cd /home/long/mesos/sbin
./mesos-slave --master=192.168.1.203:5050 --quiet &
>> sudo chmod 777 startmesosslave

10、设定无密码启动

>> sudo visudo
long ALL=NOPASSWD:/usr/local/bin/startmesosslave

11、安装Pymesos

>> git clone https://github.com/douban/pymesos.git
>> cd pymesos
>> sudo apt-get install python-pip
>> sudo python setup.py install

12、安装Dpark

>> sudo apt-get install git libzmq-dev python-pip python-dev
>> git clone https://github.com/douban/dpark.git
>> cd dpark
>> sudo apt-get install python-psutil psutils
>> sudo python setup.py install

13、配置ssh无密码登录

>> scp long@192.168.1.203:/home/long/.ssh/authorized_keys ~/.ssh/
>> sudo chmod 600 ~/.ssh/authorized_keys

14、修改/etc/hosts

>> sudo emacs /etc/hosts
192.168.1.201    pc1
192.168.1.202    pc2
192.168.1.203    pc3
192.168.1.204    pc4
192.168.1.211    cn001
192.168.1.212    cn002
192.168.1.213    cn003
192.168.1.214    cn004
192.168.1.215    cn005
192.168.1.216    cn006
192.168.1.217    cn007
192.168.1.218    cn008
192.168.1.219    cn009
192.168.1.220    cn010
192.168.1.221    cn011
192.168.1.222    cn012
192.168.1.241    snc001
192.168.1.242    snc002
192.168.1.243    snc003
192.168.1.244    snc004
192.168.1.210    client

15、测试Beansdb

>> sudo emacs dbtest.py
from dbclient import Beansdb

BEANSDBCFG = {
    "192.168.1.203:7900": range(16)
}

db = Beansdb(BEANSDBCFG, 16)
for x in range(1, 10):
    db.set(str(x), str(x**2))
for x in range(1, 10):
    print(db.get(str(x)))
>> python dbtest.py
>> 1
>> 4
>> 9
>> 16
>> 25
>> 36
>> 49
>> 64
>> 81

16、测试Dpark

>> emacs dparktest.py
import time
start = time.clock()

from dpark import DparkContext
from random import random
dpk = DparkContext()
count = dpk.accumulator(0)


def random_once(*args, **kwrgs):
    x = random() * 2 - 1
    y = random() * 2 - 1
    if x * x + y * y < 1:
        count.add(1)

N = 100000

dpk.parallelize(range(0, N), 50).foreach(random_once)
print("PI is roughly: " + str(4.0 * count.value / N))

end = time.clock()
print(end - start)
>> python dparktest.py
>> python dparktest.py -m mesos://192.168.1.203:5050

17、测试MooseFS

>> emacs mfstest.py
from dpark import DparkContext
dpk = DparkContext()
f = dpk.textFile("/mnt/mfs/lastfm-dataset-1K/userid-profile.tsv")
print(f.count())
>> python mfstest.py -m mesos://192.168.1.203:5050

Ubuntu14.04部署分布式计算框架Dpark(2)

在pc1上

1、安装KVM,参考文章 https://codinglonglong.github.io/posts/ubuntuan-zhuang-kvmxu-ni-ji/

2、配置KVM桥接

>> sudo apt-get install uml-utilities
>> sudo emacs /etc/network/interfaces
auto eth0
iface eth0 inet manual

auto tap0
iface tap0 inet manual
up ifconfig $IFACE 0.0.0.0 up
down ifconfig $IFACE down
tunctl_user long

auto br0
iface br0 inet static
bridge_ports eth0 tap0
address 192.168.1.201
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
dns-nameserver 202.98.198.167

3、删除virbr0,参考文章 https://codinglonglong.github.io/posts/kvmshan-chu-virbr0/

4、重启系统

5、启动virt-manager,安装计算节点cn001,使用Ubuntu Server 14.04.3系统,1个CPU,2G内存,10G硬盘。

Ubuntu修改root密码

Ubuntu修改root密码的方法如下:

long@pc3:/usr/local/lib/python2.7/dist-packages$ sudo passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
long@pc3:/usr/local/lib/python2.7/dist-packages$ su - root
Password:
root@pc3:~# ls