[Trouble shooting / Linux] The volume "Filesystem root" has only 0 bytes disk space remaining.

• For my case, used % of /var was 100%. Especially, /var/log/kern.log was almost 20GB and /var/log/syslog was also large.
  
• If you type as follows, you can find the log files which are very huge size.

$ ls -lSh /var/log

Solution

$ cd /var/log

$ sudo apt clear

$ sudo  apt autoremove --purge

• Set rotate number as 1 (e.g.,rotate 4 --> 1)

$ vi /etc/logrotate.conf 

# keep 4 weeks worth of backlogs
(before) rotate 4
(after) rotate 1

• Delete old backup and log files such as *.log.3.
• Empty (NOT REMOVE!!!) kern.log and syslog

$ sudo su
$ > kern.log
$ > syslog
$ > (dpkg.log, wtmp, lastlog and so on)
$ exit

[Kaldi, Trouble shooting] 읽고 정리할 것

Reference:

https://groups.google.com/forum/#!topic/kaldi-help/tzyCwt7zgMQ

http://white.ucc.asn.au/Kaldi-Notes/tidigits/data_prep



Trouble shooting reference

1. Question about the "trials" file for ivector-compute-dot-products
2. Scoring i-vector
https://groups.google.com/forum/#!topic/kaldi-help/38r7FzlzovE

3. Creating utt2spk file
https://sourceforge.net/p/kaldi/discussion/1355348/thread/44279e5a/?limit=25#3880

[Kaldi, Trouble shooting] Sun GrindEngine (Open GridEngine) error

Reason: Sun GridEngine (Open GridEngine) is not installed on your computer

Solution:
Change the cmd.sh "queue.pl" as "run.pl"


Reference: Parallelization in Kaldi
http://www.danielpovey.com/kaldi-docs/queue.html


==============================================================
[Clipping]

1. How to setup open Grind Engine Cluster using Ubuntu
http://linoxide.com/ubuntu-how-to/setup-open-grid-engine-cluster-ubuntu/

2. http://www.socher.org/index.php/Main/HowToInstallSunGridEngineOnUbuntu

3. Korean
http://blog.syszone.co.kr/674

[Linux,Trouble shooting] /bin/sh^M: bad interpreter: No such file or directory

When you create a script file on Windows and then run it on Linux,
an error occurs as follows:

     /bin/sh^M: bad interpreter: No such file or directory

It is because the 'new line character' is different. 


< Solution >

1. vi -b
: go to the binary mode of the vi
-> Then, you could see the character ^M
-> Delete this.


2. dos2unix   filename.txt
: Modify the file with the command on Linux console




============================================================================
Windows 에서 script file을 작성하고 Linux에서 file을 실행하면 아래와 같은 에러가 발생하는데, 이는 개행문자 (줄바꿈 문자)가 다르기 때문이다

     /bin/sh^M: bad interpreter: No such file or directory


< 해결 방법 >

1. vi -b
: vi 의 바이너리 모드로 들어가서 ^M 이라는 문자를 지워준다.


2. dos2unix   filename.txt
: Llinux console에서 위의 명령어로 파일을 변경한다.

[Trouble shooting] [MATLAB] Error using unique function; Error using cell/unique. Input A must be a cell array of string

Problem

If the input (A) of unique function contains non-string components, an error occurs.

Check whether the input A contains non-string components or not:

>> A = {'hello';'world';'hello';'young'};
>> unique(A(:,1))
ans = 
    'hello'
    'world'
    'young'

However if the A contains non-string components as like:

>> A = {'hello';[];'hello';'young'};

iscellstr(A(:,1))

ans =

      0

Solution

>> find(cellfun(@(x)~ischar(x),A))

ans =

      2

idxNonString = find(cellfun(@(x)~ischar(x),A));

A(idxNonString,1) = [];

unique(A(:1));

[MATLAB][Trouble shooting] MTIMES is not fully supported for integer classes.

Error using * MTIMES is not fully supported for integer classes. At least one input must be scalar. To compute elementwise TIMES, use TIMES (.*) instead.


When using a GPU, the types of variables should be same.
Like, both are double or both are integer...
-> 'Cast' the data type to double or integer

[Trouble shooting] How to delete the folders of Cygwin? (permission denied, access is denied)

Problem.

If you couldn't delete the Cygwin folder because of the access problem.



Solution.

1. Run cmd as administrator mode.

2. Go to the disk where the Cygwin is installed. (e.g., C:\)

3. Type the following commands

• takeown /r /d y /f cygwin

(Above command takes ownership recursive of the folder, without asking anything.)

• icacls cygwin /t /grant Everyone:F

(It gives full access to everyone recursively in the folder)

• rmdir /s /q cygwin

(This command deletes cygwin and subfolders)

[Trouble shooting] HTK WARNING [-2637] HeaviestMix: mix 1 in sp has v.small gConst

Problem
HHEd -A -H mix_moreA/hmmdefs -M mix_moreA 10.hedscript tiedlist
WARNING [-2637]  HeaviestMix: mix 1 in sp has v.small gConst [52.646065] in HHEd


Solution
Reference
From HTK books:

10.6 Mixture Incrementing
...
In HTK therefore, the conversion from single Gaussian HMMs to multiple mixture component HMMs is usually one of the final steps in building a system. The mechanism provided to do this is the HHEd MU command which will increase the number of components in a mixture by a process called mixture splitting. This approach to building a multiple mixture component system is extremely flexible since it allows the number of mixture components to be repeatedly increased until the desired level of performance is achieved.
The MU command has the form

MU n itemList

where n gives the new number of mixture components required and itemList defines the actual mixture distributions to modify. This command works by repeatedly splitting the ’heaviest’ mixture component until the required number of components is obtained. The ’heaviness’ score of a mixture component is defined as the mixture weight minus the number of splits involving that component that have already been carried out by the current MU command. Subtracting the number of splits discourages repeated splitting of the same mixture component. If the GCONST value of a component is more than four standard deviations smaller than the average gConst, a further adjustment is made to the ’heaviness’ score of the component in order to make it very unlikely that the component will be selected for splitting. The actual split is performed by copying the mixture component, dividing the weights of both copies by 2, and finally perturbing the means by plus or minus 0.2 standard deviations.
...
Normally, however, the number of components in all mixture distributions will be increased at the same time. Hence, a command of the form is more usual

MU 3 {*.state[2-4].mix}

It is usually a good idea to increment mixture components in stages, for example, by incrementing by 1 or 2 then re-estimating, then incrementing by 1 or 2 again and re-estimating, and so on until the required number of components are obtained. This also allows recognition performance to be monitored to find the optimum.
One final point with regard to multiple mixture component distributions is that all HTK tools ignore mixture components whose weights fall below a threshold value called MINMIX (defined in HModel.h). Such mixture components are called defunct. Defunct mixture components can be prevented by setting the -w option in HERest so that all mixture weights are floored to some level above MINMIX. If mixture weights are allowed to fall below MINMIX then the corresponding Gaussian parameters will not be written out when the model containing that component is saved. It is possible to recover from this, however, since the MU command will replace defunct mixtures
before performing any requested mixture component increment.

[Trouble shooting] HTK ERROR [+7031] PutTransMat: Row N of transition mat sub = 1.xxx

Problem
HERest -C src/ConfigHVite -I lists/train.phonemlf -t 250.0 150.0 1000.0 -S train.mfcc.list -H hmm0/macros -H hmm0/hmmdefs -M hmm1 lists/monophones1
Saving hmm's to dir hmm1
ERROR [+7031] PutTransMat: Row 4 of transition mat sum = 1.064684
FATAL ERROR - Terminating program HERest


Solution
Too much data. Use the -p option, splitting the input and processing over several machines, then doing a separate HERest pass with -p 0 to accumulate the accumulators. Or, as above, use a smaller portion of the data. Also, make sure that the file durations are spread evenly across lists. Don't put all the long files together, mix them up with short ones.


Reference
From HTK books:
-p N This switch is used to set parallel mode operation. If p is set to a positive integer N, then
HERest will process the training files and then dump all the accumulators into a file called
HERN.acc. If p is set to 0, then it treats all file names input on the command line as the
names of .acc dump files. It reads them all in, adds together all the partial accumulations
and then re-estimates all the HMM parameters in the normal way.

======================================================
This problem could appear when the sum of the specific row of transition matrix is not 1.

It could be possible when the # of training data is too much.

--> Split the training script file and run HERest as parallel mode using 'p' option.

e.g.,

HERest -S trnlist1 -I lab1 -H dir1/hmmdefs1 -M dir2 -p 1 hmmlist    # part1
HERest -S trnlist2 -I lab2 -H dir1/hmmdefs2 -M dir2 -p 2 hmmlist    # part2
HERest                      -H dir1/hmmdefs  -M dir2 -p 0 hmmlist  dir2/*.acc # Merging

[Trouble shooting] (HTK) ERROR [+7332] CreateInsts: Cannot have successive Tee models

Error message : 
 ERROR [+7332]  CreateInsts: Cannot have successive Tee models

Reason :

 When you have successive silence model, this kinds of error appears.

 e.g., 'sil-sil', 'sil-sp', 'sp-sp'

Reference :

 http://www.voxforge.org/home/dev/acousticmodels/linux/create/htkjulius/tutorial/monophones/step-7/comments/errore-creating-file-for-hmm6#7_1Yg71NqpoBvxz3jb21XA

[Trouble shooting] error LNK2019: unresolved external symbol _WinMain@16 referenced in function ___tmainCRTStartup

Microsoft Visual Studio 2010

THIS IS A LINK PROBLEM

• Error message
• error LNK2019: unresolved external symbol _WinMain@16 referenced in function ___tmainCRTStartup
  
  
• MSVCRTD.lib(crtexew.obj) : error LNK2019: _WinMain@16 외부 기호(참조 위치: ___tmainCRTStartup 함수)에서 확인하지 못했습니다.

• Solution
• Properties -> linker -> system -> sub system
  'Windows (/SUBSYSTEM:WINDOWS)' to 'Console(/SUBSYSTEM:CONSOLE)'
   
• 프로젝트->속성->구성 속성->링커->시스템->하위 시스템
  '창 (/SUBSYSTEM:WINDOWS)'를 '콘솔 (/SUBSYSTEM:CONSOLE)'로 바꿔 준다.

[Trouble shooting] error C4996 : 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS.

error C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instread. to disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.


This is not an error but an warning.


If you use old C standard functions Visual Studio blocks to complie because it is unsafe for security reasons.

Basically 'using more save functions' is recommended like fopen_s().


If you wanna other solution, change setups as follows :

Solution.

1. Use 'fopen_s' instead of 'fopen'.

2. Permit '_CRT_SECURE_NO_WARNINGS' by following procedures.

 Select 'Project' -> 'Properties' -> 'Configuration Properties' -> 'C/C++' -> 'Preprocessor' -> add ';_CRT_SECURE_NO_WARNINGS' in the 'Preprocessor Definitions' to turn of warnings.

[Trouble shooting] C++, LINK : fatal error LNK1123: COFF로 변환하는 동안 오류가 발생했습니다.

Microsoft Visual Studio 2010

• Error message
• LINK : fatal error LNK1123: COFF로 변환하는 동안 오류가 발생했습니다. 파일이 잘못되었거나 손상되었습니다.

• Solution
• Menu -> Project -> Project properties (Alt+F7) -> Manifest tool -> Including manifest : YES-->NO

[Trouble shooting] There is a screen on ... When you can't attach on the screen!

Problem)

I turned off by accident my computer without termination.

When I'm trying to log on my screen again, following error appears.

>> screen -r myScreenName

There is a screen on :

15735.myScreenName (Attached)

Solution)

Try detach it first with screen -d <15735> or screen -D <pid>

[Trouble shooting] 데이터를 암호화하는 동안 발생한 오류 때문에 이 세션을 종료합니다. / Because of error in data encryption, this session will end.

Problem.

원격 데스크탑으로 다른 윈도우 컴퓨터에 접속할 때,

'데이터를 암호화하는 동안 발생한 오류 때문에 이 세션을 종료합니다. 원격 컴퓨터에 다시 연결해 보십시오.'

오류 메시지가 뜨면서 원격 접속이 꺼지고, 작업하던 프로그램들이 모두 종료되었을 때.



Solution.

1. 오류 해결 툴을 이용

--> http://support.microsoft.com/kb/323497/ko#LetMeFixItMyselfAlways


2. 수동으로 레지스트리 수정

--> 아래의 경로에서 인증서, x509인증서, x509인증서 ID 삭제 후 원격 컴퓨터 재부팅.

--> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TermService\Parameters

Problem.

'Because of error in data encryption, this session will end. Please try connecting to the remote computer again.'


Solution.

1. Use tool

--> http://support.microsoft.com/kb/323497/ko#LetMeFixItMyselfAlways


2. Change registry manually

--> Delete certificate, x509 certificate, x509 certificate ID in following directory, then reboot remote computer.

--> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TermService\Parameters

Because of error in data encryption, this session will end.

[Trouble shooting / MATLAB] Data memory allocation, speed acceleration, code optimization

If you put a data into an array or any memory repeatedly,
to allocate memory is much x1000000 faster than enlarge memory every iteration.




Code example.

#1. repeatedly enlarge memory (worst case)

     DB          = struct('phoneme',phone, 'cnt', 0, 'waveform',[]);
     DB.cnt      = DB_in.cnt;
   
    for ii = 1 : DB.cnt
        DB.waveform(ii,:) = func_fix_window(DB_in.waveform{ii}, L);
    end




#2. pre-allocate memory with size (it is much faster)

    DB = struct('phoneme',phone, 'cnt',DB_in.cnt, 'waveform',zeros(DB_in.cnt,L));
   
    for ii = 1 : DB.cnt
        DB.waveform(ii,:) = func_fix_window(DB_in.waveform{ii}, L);
    end

[Trouble shooting] MATLAB, segmentation violation detected

Following error occurs when I try to run the 'code A'.

I guess the caution of the problem is '.mat file'!!!.
In the 'code A', it loads some '.mat file'.

However!! other program 'code B' updates (load & add or modify variables & save) the '.mat file' repeatedly.

I guess the error occurs because 'code A' tries to load 'incompletely save .mat file'!
If I change the '.mat file', the error does not occur.


This is just my guess.





------------------------------------------------------------------------
       Segmentation violation detected at Mon Mar 31 21:12:13 2014
------------------------------------------------------------------------

Configuration:
  MATLAB Version:   7.9.0.529 (R2009b)
  MATLAB License:   161051
  Operating System: Linux 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64
  GNU C Library:    2.15 stable
  Window System:    No active display
  Current Visual:   None
  Processor ID:     x86 Family 6 Model 10 Stepping 7, GenuineIntel
  Virtual Machine:  Java 1.6.0_12-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
  Default Encoding:  UTF-8

Fault Count: 1

Register State:
  rax = 0000000000000000   rbx = 0000000000000008
  rcx = 0000000000008000   rdx = 00007f1d140000a8
  rbp = 00007f1d1a9ea7a0   rsi = 00007f1d140000c8
  rdi = 0000000000000010   rsp = 00007f1d1a9ea748
   r8 = 0000000000000000    r9 = 0000000000000008
  r10 = 0000000000000022   r11 = 0000000000000246
  r12 = 00007f1d179b1880   r13 = 00007f1d17a4dfe0
  r14 = 0000000000000001   r15 = 00007f1d1a9ea888
  rip = 00007f1d293f6df4   flg = 0000000000010202

Stack Trace:
  [0] libc.so.6:0x00007f1d293f6df4(0x7f1d1a9ea898, 0x7f1d200ffc64, 0x7f1d2008e95d, 0)
  [1] libmat.so:get_fields_attribute(int, bool&, int&, char**&)(0x7f1d1a9ea8d0, 0x7f1d1a9ea898, 0x7f1d17a4c0e0 "struct", 0x2316c0fe) + 497 bytes

[Trouble Shooting] C 재정의 오류, 헤더파일 중복 방지, #if~#endif, #pragma once

Problem.

• C에서 프로그래밍을 하다가 헤더파일을 중복으로 include 한 경우, 컴파일時 '재정의' 오류가 발생한다.

Solution.

• 헤더파일에 맨 위에 아래의 명령어를 추가하여 헤더파일이 중복으로 컴파일되는 것을 방지한다.

1) #ifndef ~ #endif

Example.
#ifndef MY_HEADER_H   // MY_HEADER_H가 선언되어 있지 않으면

#define MY_HEADER_H  // '여기부터 #endif 이전 라인까지 포함시켜라'는 전처리 지시어.
//..code..

#endif




2) #pragma once

단 한번만 처리를 하고, 다음부터는 같은 파일의 경우 컴파일러가 헤더 파일을 열지도 않는다.
컴파일 속도는 빨라지나, 일부 컴파일러에서는 지원이 안 될 수도 있으므로 안정성과 범용성 측면에 있어서는 #ifndef ~ #endif를 사용하는 것을 추천함.

[Trouble shooting] c에서 binary 파일을 끝까지 읽지 못할 경우! fread

c에서 binary 파일을 fopen으로 열고, fread를 이용하여 데이터를 로드하였는데,
데이터를 끝까지 읽어들이지 못하고 중간에 멈추었다.


...
short inbuf;
FILE *fin = fopen("hello.pcm","r");
while( fread(&inbuf, sizeof(short),1, fin) ){
}
...


근래들어 MATLAB만 사용하다가 오랫만에 C를 사용해서 그런가
이런 말도 안되는 사소한 문제에 빠져서 헤매다가 드디어 해결!!


FILE *fin = fopen("hello.pcm","rb");

fopen() 함수는 기본 file mode가 TEXT 상태이다!!!
그렇기 때문에 모드를 생략하면 파일을 text로 open하는데,
내가 열려는 파일은 BINARY 파일이였던 것이다.

MATLAB에서는 알아서 파일에 맞는 모드로 파일을 열었었는데,
꼼꼼히 설정해 줘야 하는 C는 나에게 불만을 마구 표출하며 골탕먹였다.

MATLAB에 너무 길들여졌더니 이러한 실수나 하고...OTL



참고. FILE *fopen(const char *filename, const char *mode);

<file mode 종류>

택1, 생략 불가
r  : 읽기 위해 open. 기존에 파일이 존재하여야 함. 없으면 에러.
w : 쓰기 위해 open. 없으면 생성. 존재하면 파일 삭제 후 다시 생성.
a : 추가하기 위해 open. 없으면 생성, 존재하면 추가.
r+ : 읽기/쓰기 위해 open. 기존에 파일이 존재하여야 함.
w+: 읽기/쓰기 위해 open. 없으면 생성, 존재하면 삭제 후 다시 생성.
a+ : 읽기/추가하기 위해 open. 없으면 생성, 존재하면 추가.

택1, 생략시 t 모드
b : 기계어 상태로 open
t : 텍스트 상태로 open

[Trouble shooting / MATLAB] save '-v7.3'

When I saved large size structure, problem comes up.

Warning : Variable 'structure name' cannot be saved to a MAT-file whose version is older than 7.3. To save this variable, use the -v7.3 switch.

- With the -v7.3 flag, you could store more than 2GB variables with compression.

- Without the -v7.3 flag, there is no compression and you could not store large variables.

I clipped the documents from the MATLAB help documents.

Version Compatibility Options

The following table lists version compatibility options. These options enable you to save your workspace data to a MAT-file that can then be loaded into an earlier version of MATLAB software. The resulting MAT-file supports only those data items and features that were available in this earlier version of MATLAB. (See the second table below for what is supported in each version.)

version Option	Use When Running ...	To Save a MAT-File That You Can Load In ...
-v7.3	Version 7.3 or later	Version 7.3 or later
-v7	Version 7.3 or later	Versions 7.0 through 7.2 (or later)
-v6	Version 7 or later	Versions 5 and 6 (or later)
-v4	Version 5 or later	Versions 1 through 4 (or later)

The default version option is the value specified in the Preferences dialog box. Select File > Preferences in the Command Window, click General, and then MAT-Files to view or change the default.
The next table shows what data items and features are supported in different versions of MATLAB. You can use this information to determine which of the version compatibility options shown above to use.

MATLAB Versions	Data Items or Features Supported
4 and earlier	Support for 2D double, character, and sparse
5 and 6	Version 4 capability plus support for ND arrays, structs, and cells
7.0 through 7.2	Version 6 capability plus support for data compression and Unicode character encoding
7.3 and later	Version 7.2 capability plus support for data items greater than or equal to 2GB

Remarks

When using the -regexp switch, save considers all variables in the argument list, with the exception of the optional filename and structure name variables, to be regular expressions. The filename, if specified, is always the first argument in the argument list, provided that this argument is a variable name. The structure name, if specified, is always the first argument following the -struct keyword, provided that the argument list includes that keyword.
When working on 64-bit platforms, you can have data items in your workspace that occupy more than 2 GB. To save data of this size, you must use the HDF5-based version of the MATLAB MAT-file. Use the –v7.3 option to do this:

save -v7.3 myfile v1 v2

If you are running MATLAB on a 64-bit computer system and you attempt to save a variable that is too large for a version 7 (or earlier) MAT-file, that is, you save without using the -v7.3 option, MATLAB skips that variable during the save operation and issues a warning message to that effect.
If you are running MATLAB on a 32-bit computer system and attempt to load a variable from a -v7.3 MAT-file that is too large to fit in 32–bit address space, MATLAB skips that variable and issues a warning message to that effect.
MAT-files saved with compression and Unicode encoding cannot be loaded into versions of MATLAB prior to MATLAB Version 7.0. If you save data to a MAT-file that you intend to load using MATLAB Version 6 or earlier, you must specify the -v6 option when saving. This disables compression and Unicode encoding for that particular save operation.
If you want to save to a file that you can then load into a Version 4 MATLAB session, you must use the -v4 option when saving. When you use this option, variables that are incompatible with MATLAB Version 4 are not saved to the MAT-file. For example, ND arrays, structs, cells, etc. cannot be saved to a MATLAB Version 4 MAT-file. Also, variables with names that are longer than 19 characters cannot be saved to a MATLAB Version 4 MAT-file.