Tokushima University / Educator and Researcher Directory --- Nishimura, Ryota

Personal Web Page

https://researchmap.jp/sayonari (researchmap)

https://orcid.org/0000-0002-2948-530X (ORCID)

Field of Study

○	Informatics

Subject of Study

○	Speech information processing (spoken dialog system, interactive interface, human interface, speech recognition, multimodal interaction)

Book / Paper

Book:

1.	Kenji Kita, Ryota Nishimura and Kazuyuki Matsumoto : Introduction to Data Preprocessing in AI and Machine Learning, --- Getting Started with Python ---, Kagakujyoho Shuppan Co., Ltd., Tokyo, Jun. 2021. (Summary) Around 2010, the term "Big Data" began to rapidly permeate the world around us. At the same time, the construction of intelligent AI systems based on machine learning from big data has even become a boom these days. This book focuses on the pre-processing of data, which is an inevitable part of building such machine learning systems, and although the primary focus is on machine learning systems such as AI, the various techniques described in this book can also be used effectively in data mining and data analysis to acquire useful knowledge from big data. However, the various techniques described in this book can also be used effectively in data mining and data analysis to obtain useful knowledge from big data.The term "big data" is pleasant to the ears and may sound like a treasure trove, but in fact big data is a mixture of cobbles and stones, a mixture of treasure and garbage. In fact, big data is a mixture of treasure and trash, and the larger the scale of the data, the more trash there is, and the more difficult it becomes to extract the treasure from it. In the field of computer science, there is a saying, "Garbage In, Garbage Out" (or GIGO for short). To build a good machine learning system, it is important to eliminate garbage from data and process the data into a form that is easy to learn, which is the essence of preprocessing.Recently, many frameworks and libraries for machine learning such as TensorFlow and Keras are available free of charge, and using these frameworks and libraries may seem to lower the threshold for programming. Using publicly available datasets for machine learning, anyone can easily create an AI system. However, there is a big problem when building your own system using your own data set. As is often said, real-world data is dirty. If you use the data as it is, that is exactly what you will get.If we use the data as it is, we will end up in a "Garbage In, Garbage Out" situation. Whether or not a useful AI system can be built from real data depends on the success of pre-processing. According to one theory, in the actual construction of AI and machine learning systems, 60-80% of the time spent by engineers is spent on data collection and preprocessing.This book focuses on preprocessing techniques, which have not been sufficiently covered in conventional books on machine learning, and aims to be a practical book where readers can understand the techniques not only by explaining them but also by actually running programs. This book has a sister volume, "Practical Edition", which is planned to be published in the near future. In "Practical Edition", more advanced preprocessing techniques and preprocessing techniques for media data such as text, images, sound, and music are explained. (Keyword) artificial intelligence / machine learning / Python / Preprocessing / data analysis / data mining / Google Colab / Big Data / Feature Selection / Dimensionality Reduction / NumPy / pandas / scikit-learn / matplotlib / TensorFlow / Keras / Data Cleaning / Feature Extraction / Scaling / Standardization / normalization / Normalization / Data Cleansing / Data Augmentation / Binning / Discretization / Outlier / IQR / Boxplot / Smirnov-Grubbs' test / Missing Value / List-wise Case Deletion / Pair-wise Case deletion / Single Imputation / Multiple Imputation / Filter Method / Chi-squared Test / analysis of variance / Pearson Correlation Coefficient / PCA / NMF / CCA / LDA / Kernel PCA / t-SNE / UMAP / MDS / self-organizing map / autoencoder / SMOTE
2.	Norihide Kitaoka, nakagawa takuma, Ryota Nishimura, Ishiguro Yoshio, Kojima Shin'ichi and Ohsuga Shin : A Multimodal Control System for Autonomous Vehicles Using Speech, Gesture, and Gaze Recognition, Walter de Gruyter & Co., 2020. (Summary) The development of autonomous vehicles has recently become the focus of much research. One area of investigation is the development of systems that will allow people to easily direct autonomous vehicles without any special technical skills or training. Human-machine interface technologies are likely to be used to achieve this objective. In this study, we propose an intuitive, multimodal interface system that uses speech, gesture, and gaze recognition to transmit user commands to the vehicle. We designed the multimodal understanding and dialog control components of the interface system separately, using finite-state transducers (FSTs), a technique which is also used to control conventional automated dialog systems. Our multimodal understanding and dialog control components can be thought of as a cascade of two separate transducers, but since cascading transducers can be combined into one transducer, our system is actually driven by a single FST. During trials, our proposed interface system allowed users to successfully direct and operate an autonomous vehicle using only speech, head movement, and gaze direction. (Link to Publication Site) ● Publication site (DOI): 10.1515/9783110669787-007 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1515/9783110669787-007 (DOI: 10.1515/9783110669787-007)
3.	Takahiro Uchiya, Ryota Nishimura, Takahiro Hirano and Masaru Sakurai : Design of Reminiscence Therapy System for Elderly People with Dementia, Springer, Nov. 2018. (Summary) 近年，認知症患者の数が増加し，介護者の負担が増大している．しかし，認知症の治療法は確立されておらず，その進行を制御することが治療の目的となっている．認知症の進行を抑制する1つの手段として，介護者と認知症患者との対話による回想法を使用する．しかし，この方法では，介護者は何度も対話を行う必要があり，介護者にとって負担となるため，介護者の代わりにMMDAgentという音声対話システムを用いて回想法対話の支援を行う方法を提案する．これによって，介護者の負担が軽減される．本稿では，提案されたシステムの設計を示す． (Link to Publication Site) ● Publication site (DOI): 10.1007/978-3-319-69811-3_76 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1007/978-3-319-69811-3_76 (DOI: 10.1007/978-3-319-69811-3_76)
4.	太田一穂, 岡嶋和弘, Ryota Nishimura and 樋山淳 : ここまで作れる! Raspberry Pi 実践サンプル集, マイナビ出版, Jul. 2017. (Keyword) Raspberry Pi
5.	Takeshi Morita, Yu Sugawara, Ryota Nishimura and Takahira Yamaguchi : Integrating Symbols and Signals Based on Stream Reasoning and ROS, Springer, Phuket, Thailand, Aug. 2016. (Summary) 実践的知能アプリケーション開発プラットフォームであるPRINTEPS(PRactical INTElligent aPplicationS)を開発した．本稿では，ストリーム推論とROS(Robot Operating System)を用いたイベント検出のためのPRINTEPSの応用と，知識処理による画像センシングの統合について述べる．このプラットフォームにおいて，我々はロボットカフェ内のロボットの動作をデモンストレーションし，ルールセットを変更することによって簡単にロボットカフェを実現・適用できることを示す． (Keyword) ROS / Stream Reasoning / PRINTEPS / Ontology
6.	Yuki Todo, Ryota Nishimura, Kazumasa Yamamoto and Seiichi Nakagawa : Development and evaluation of spoken dialog systems with one or two agents through two domains, Springer, Berlin Heidelberg, Sep. 2013. (Summary) Almost all current spoken dialog systems treat dialog as that where a single user talks to an agent. On the other hand, we set out to investigate a multiparty dialog system that deals with two agents and a single user. We developed a three person (one user and two agents) and a two person (one user and one agent) dialog system to consider the same dialog tasks, that is, Which do you prefer, udon or ramen (Japanese noodle or Chinese noodle)? and Which do you want to travel to Hokkaido or Okinawa (snowy region or tropical region)? and compared them with respect to user behavior and satisfaction. According to the results of the experiments, the three person dialog system performed better in terms of lively conversation, and user can talk with the agents more like chatting. (Keyword) spoken dialog system, / multi-party dialogue / two agents / chat (Link to Publication Site) ● Publication site (DOI): 10.1007/978-3-642-40585-3_24 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1007/978-3-642-40585-3_24 (DOI: 10.1007/978-3-642-40585-3_24)
7.	Ryota Nishimura, Norihide Kitaoka and Seiichi Nakagawa : A spoken dialog system for chat-like conversations considering response timing, Springer, Sep. 2007. (Summary) If a dialog system can respond to a user as naturally as a human, the interaction will be smoother. In this research, we aim to develop a dialog system by emulating the human behavior in a chat-like dialog. In this paper, we developed a dialog system which could generate chat-like responses and their timing using a decision tree. The system could perform collaborative completion, aizuchi (back-channel) and so on. The decision tree utilized the pitch and the power contours of users utterance, recognition hypothesis, and response preparation status of the response generator, at every time segment as features to generate response timing.

Academic Paper (Judged Full Paper):

1.	Kiuchi Keita, XIN KANG, Ryota Nishimura, Sasayama Manabu and Kazuyuki Matsumoto : Predicting Physical and Mental Health Status One Month Later through Interview-Based Stress Assessment, Industrial Health, Vol.62, No.4, 237-251, (Summary) This study conducted an interview-based stress evaluation that considered the psychosocial models of work stress and verified the evaluations predictive validity. A four-stage assessment comprising a pre-survey, pre-interview questionnaire, stress assessment interview, and post-survey after one month was conducted with 50 Japanese workers. Additionally, 16 occupational health professionals provided stress evaluations based on recorded interview videos. Variables based on intraclass correlation coefficients (ICCs) were computed in multiple ways to compare the agreement among the evaluators. The generalized estimating equation (GEE) was conducted to evaluate the prediction models. The overall ICC among the evaluators was 0.58. The GEE revealed that the mean score of the evaluators in the interview-based stress evaluation significantly predicted psychological symptoms (β =2.02, p=0.019), burnout (β =0.77, p<0.001), and well-being (β =0.64, p=0.007) one month later, even after adjusting for the self-reported stress levels measured in the pre-survey. The predictive validity of the proposed interview-based stress evaluation was confirmed. Although there are several challenges in standardizing this evaluation, semi-structured interviews are an effective tool for understanding work stress. (Keyword) burnout / chronic stress / interview-based assessment / job stress / stress assessment / well-being (Link to Publication Site) ● Publication site (DOI): 10.2486/indhealth.2023-0144 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.2486/indhealth.2023-0144 (DOI: 10.2486/indhealth.2023-0144)
2.	Kohei Torii, Ryota Nishimura and Eiichi Honda : Segmentation of Mandibular Canal on Dental Cone Beam CT Images with AI Development Support Software for Medical Images, Dental Radiology, Vol.64, No.1, 11-19, 2024. (Keyword) medical image / artificial intelligence / Segmentation (Tokushima University Institutional Repository) ● Metadata: 119604 (Tokushima University Institutional Repository: 119604)
3.	Ryota Nishimura, Takaaki Uno, Taiki Yamamoto, Kengo Ohta and Norihide Kitaoka : Detection of Arbitrary Wake Words by Coupling a Phoneme Predictor and a Phoneme Sequence Detector, APSIPA Transactions on Signal and Information Processing, 2024. (Summary) Most wake word (WW) detection systems used in smartphones and smart speakers only detect specific, predefined WWs such as Hey, Siri or OK, Google. To build such a system, a large speech corpus consisting of many examples of the selected WWs must be collected to train the model. If we want the device to detect a different WW, collection of a new speech corpus and re-training of the model are required. In this study, we propose a system which is capable of detecting any chosen WW without additional model training or a corpus of WW utterances, allowing users to select and use their preferred WW. Our system consists of a phoneme predictor (PP) and a phoneme sequence detector (PSD). The PP predicts phoneme sequences using acoustic features of the input speech, and outputs phoneme probability distributions. The acoustic models in the PP are trained using the Connectionist Temporal Classification (CTC) loss criterion. The PSD takes the output of the PP as input, and predicts the probability of whether or not the WW has been input. In our evaluation experiments, we performed six-phoneme WW detection. Our results showed that the proposed method achieved 90% WW detection accuracy. (Keyword) wake word / CTC / end-to-end modeling / phoneme sequence detector (Link to Publication Site) ● Publication site (DOI): 10.1561/116.20240014 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1561/116.20240014 (DOI: 10.1561/116.20240014)
4.	Mori Daiki, Ohta Kengo, Ryota Nishimura, Ogawa Atsunori and Norihide Kitaoka : Recognition of target domain Japanese speech using language model replacement, EURASIP Journal on Audio, Speech, and Music Processing, Vol.2024, No.1, 1-14, 2024. (Summary) End-to-end (E2E) automatic speech recognition (ASR) models, which consist of deep learning models, are able to perform ASR tasks using a single neural network. These models should be trained using a large amount of data; however, collecting speech data which matches the targeted speech domain can be difficult, so speech data is often used that is not an exact match to the target domain, resulting in lower performance. In comparison to speech data, in-domain text data is much easier to obtain. Thus, traditional ASR systems use separately trained language models and HMM-based acoustic models. However, it is difficult to separate language information from an E2E ASR model because the model learns both acoustic and language information in an integrated manner, making it very difficult to create E2E ASR models for specialized target domain which are able to achieve sufficient recognition performance at a reasonable cost. In this paper, we propose a method of replacing the language information within pre-trained E2E ASR models in order to achieve adaptation to a target domain. This is achieved by deleting the implicit language information contained within the ASR model by subtracting the source-domain language model trained with a transcription of the ASRs training data in a logarithmic domain. We then integrate a target domain language model through addition in the logarithmic domain. This subtraction and addition to replace of the language model is based on Bayes theorem. In our experiment, we first used two datasets of the Corpus of Spontaneous Japanese (CSJ) to evaluate the effectiveness of our method. We then we evaluated our method using the Japanese Newspaper Article Speech (JNAS) and CSJ corpora, which contain audio data from the read speech and spontaneous speech domain, respectively, to test the effectiveness of our proposed method at bridging the gap between these two language domains. Our results show that our proposed language model replacement method achieved better ASR performance than (Keyword) End-to-end speech recognition / Implicit language information / Language model replacement (Link to Publication Site) ● Publication site (DOI): 10.1186/s13636-024-00360-8 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1186/s13636-024-00360-8 (DOI: 10.1186/s13636-024-00360-8)
5.	Keita Kiuchi, XIN KANG, Ryota Nishimura, Manabu Sasayama and Kazuyuki Matsumoto : Causal Effects of High Stress Assessed via Interviews on Mental and Physical Health: Toward Computer Agent-Driven Stress Assessment, Journal of Occupational and Environmental Medicine, 2024. (Summary) This study investigated the causal effect of high stress assessment via an interview on the mental and physical health of workers one month later. Stress assessment interviews and feedback were conducted with 50 Japanese workers. In addition to the interviewer, two occupational health professionals assessed participants' stress based on recordings. The average treatment effect was estimated by propensity score matching. High stress, according to the interview-based assessment, had a significant negative causal effect on self-reported well-being one month later (95% confidence interval: -3.02, -1.10). In addition, no effect of high stress on stress load, mental and physical symptoms, or burnout was observed. This study provides important insights into the prognosis of individuals who were assessed through interviews to have high stress. The findings are expected to help automate stress assessments using computer agents. (Link to Publication Site) ● Publication site (DOI): 10.1097/JOM.0000000000003117 (Link to Search Site for Scientific Articles) ● PubMed @ National Institutes of Health, US National Library of Medicine (PMID): 38603579 ● Search Scopus @ Elsevier (PMID): 38603579 ● Search Scopus @ Elsevier (DOI): 10.1097/JOM.0000000000003117 (DOI: 10.1097/JOM.0000000000003117, PubMed: 38603579)
6.	Tsubasa Miyashita, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : Extraction of Lifestyle Habits Described in Blogs of Diabetics, IEEJ Transactions on Electronics, Information and Systems, Vol.142, No.10, 1144-1155, 2022. (Summary) Tackling diabetes, an increasingly common lifestyle-related illness, requires information on patients lifestyle and habits. Blogs maintained by patients afflicted with the incurable illness may be useful for analyzing how lifestyle affects health-status. In this paper, we propose an intrinsic expression extractor that extracts keywords related to lifestyle and health-status from blogs of patients diagnosed with type-2 diabetes. To counter class imbalance and add to the corpus, the proposed method tags each keyword with information based on cue-words extracted from manually tagged data. The named-entity recognition (NER) for the extracted keywords uses a bidirectional gated recurrent unit neural network (BiGRU) and was evaluated for accuracy by cross-validation. We obtained F1-score of approximately 0.76. Although the accuracy of extraction can further be improved, the novel approach has applications in analyzing and improving the lifestyle of diabetes-afflicted patients. (Keyword) disease-fighting weblogs / named entity recognition / natural language processing (Link to Publication Site) ● Publication site (DOI): 10.1541/ieejeiss.142.1144 (Link to Search Site for Scientific Articles) ● CiNii @ National Institute of Informatics (CRID): 1390293633437552768 ● Search Scopus @ Elsevier (DOI): 10.1541/ieejeiss.142.1144 (DOI: 10.1541/ieejeiss.142.1144, CiNii: 1390293633437552768)
7.	Meiko Fukuda, Ryota Nishimura, Hiromitsu Nishizaki, Koharu Horii, Yurie Iribe, Kazumasa Yamamoto and Norihide Kitaoka : A new speech corpus of super-elderly Japanese for acoustic modeling, Computer Speech & Language, Vol.77, 101424, 2022. (Summary) The development of accessible speech recognition technology will allow the elderly to more easily access electronically stored information. However, the necessary level of recognition accuracy for elderly speech has not yet been achieved using conventional speech recognition systems, due to the unique features of the speech of elderly people. To address this problem, we have created a new speech corpus named EARS (Elderly Adults Read Speech), consisting of the recorded read speech of 123 super-elderly Japanese people (average age: 83.1), as a resource for training automated speech recognition models for the elderly. In this study, we investigated the acoustic features of super-elderly Japanese speech using our new speech corpus. In comparison to the speech of less elderly Japanese speakers, we observed a slower speech rate and extended vowel duration for both genders, a slight increase in fundamental frequency for males, and a slight decrease in fundamental frequency for females. To demonstrate the efficacy of our corpus, we also conducted speech recognition experiments using two different acoustic models (DNN-HMM and transformer-based), trained with a combination of data from our corpus and speech data from three conventional Japanese speech corpora. When using the DNN-HMM trained with EARS and speech data from existing corpora, the character error rate (CER) was reduced by 7.8% (to just over 9%), compared to a CER of 16.9% when using only the baseline training corpora. We also investigated the effect of training the models with various amounts of EARS data, using a simple data expansion method. The acoustic models were also trained for various numbers of epochs without any modifications. When using the Transformer-based end-to-end speech recognizer, the character error rate was reduced by 3.0% (to 11.4%) by using a doubled EARS corpus with the baseline data for training, compared to a CER of 13.4% when only data from the baseline training corpora were used. (Tokushima University Institutional Repository) ● Metadata: 117813 (Link to Publication Site) ● Publication site (DOI): 10.1016/j.csl.2022.101424 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1016/j.csl.2022.101424 (Tokushima University Institutional Repository: 117813, DOI: 10.1016/j.csl.2022.101424)
8.	Ryota Nishimura, Raita Mori and Norihide Kitaoka : Input utterance complementation method by anaphora resolution for spontaneous utterances on spoken dialog systems, Transactions of the Japanese Society for Artificial Intelligence, Vol.37, No.3, 1-13, 2022. (Summary) In this study, we propose a method for generating response utterances which take into account contexts and topics of the dialog by complementing omitted words such as subjects in the input utterances of dialog systems. In order to complement omitted words in the input utterances, an automatic anaphora resolution based on the centering theory is performed. To achieve highly accurate anaphora resolution, we also performed spoken-to-written style con- version based on sequence-to-sequence model using LSTM as a preprocessing. The results of evaluation experiments using NUCC, the Nagoya University Conversation Corpus showed that our proposed complementation method works robustly against errors in spoken-to-written style conversion. (Keyword) Spoken Dialog system / Anaphora Resolution / Centering Theory / Spoken-to-written Style Conversion (Link to Publication Site) ● Publication site (DOI): 10.1527/tjsai.37-3_ids-f (Link to Search Site for Scientific Articles) ● CiNii @ National Institute of Informatics (CRID): 1390010457669588864 ● Search Scopus @ Elsevier (DOI): 10.1527/tjsai.37-3_ids-f (DOI: 10.1527/tjsai.37-3_ids-f, CiNii: 1390010457669588864)
9.	Byambadorj Zolzaya, Ryota Nishimura, Altangerel Ayush, Ohta Kengo and Norihide Kitaoka : Text-to-speech system for low-resource language using cross-lingual transfer learning and data augmentation, EURASIP Journal on Audio, Speech, and Music Processing, Vol.2021, No.42, 1-20, 2021. (Summary) Deep learning techniques are currently being applied in automated text-to-speech (TTS) systems, resulting in significant improvements in performance. However, these methods require large amounts of text-speech paired data for model training, and collecting this data is costly. Therefore, in this paper, we propose a single-speaker TTS system containing both a spectrogram prediction network and a neural vocoder for the target language, using only 30 min of target language text-speech paired data for training. We evaluate three approaches for training the spectrogram prediction models of our TTS system, which produce mel-spectrograms from the input phoneme sequence: (1) cross-lingual transfer learning, (2) data augmentation, and (3) a combination of the previous two methods. In the cross-lingual transfer learning method, we used two high-resource language datasets, English (24 h) and Japanese (10 h). We also used 30 min of target language data for training in all three approaches, and for generating the augmented data used for training in methods 2 and 3. We found that using both cross-lingual transfer learning and augmented data during training resulted in the most natural synthesized target speech output. We also compare single-speaker and multi-speaker training methods, using sequential and simultaneous training, respectively. The multi-speaker models were found to be more effective for constructing a single-speaker, low-resource TTS model. In addition, we trained two Parallel WaveGAN (PWG) neural vocoders, one using 13 h of our augmented data with 30 min of target language data and one using the entire 12 h of the original target language dataset. Our subjective AB preference test indicated that the neural vocoder trained with augmented data achieved almost the same perceived speech quality as the vocoder trained with the entire target language dataset. Overall, we found that our proposed TTS system consisting of a spectrogram prediction network and a PWG neural vocoder was able to achieve reasonable performance using only 30 min of target language training data. We also found that by using 3 h of target language data, for training the model and for generating augmented data, our proposed TTS model was able to achieve performance very similar to that of the baseline model, which was trained with 12 h of target language data. (Tokushima University Institutional Repository) ● Metadata: 116894 (Link to Publication Site) ● Publication site (DOI): 10.1186/s13636-021-00225-4 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1186/s13636-021-00225-4 (Tokushima University Institutional Repository: 116894, DOI: 10.1186/s13636-021-00225-4)
10.	Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Response Type Selection for Chat-like Spoken Dialog Systems Based on LSTM and Multi-task Learning (accepted), Speech Communication, Vol.133, No.2021, 23-30, 2021. (Summary) We propose a method of automatically selecting appropriate responses in conversational spoken dialog systems by explicitly determining the correct response type that is needed first, based on a comparison of the users input utterance with many other utterances. Response utterances are then generated based on this response type designation (back channel, changing the topic, expanding the topic, etc.). This allows the generation of more appropriate responses than conventional end-to-end approaches, which only use the users input to directly generate response utterances. As a response type selector, we propose an LSTM-based encoder-decoder framework utilizing acoustic and linguistic features extracted from input utterances. In order to extract these features more accurately, we utilize not only input utterances but also response utterances in the training corpus. To do so, multi-task learning using multiple decoders is also investigated. To evaluate our proposed method, we conducted experiments using a corpus of dialogs between elderly people and an interviewer. Our proposed method outperformed conventional methods using either a point-wise classifier based on Support Vector Machines, or a single-task learning LSTM. The best performance was achieved when our two response type selectors (one trained using acoustic features, and the other trained using linguistic features) were combined, and multi-task learning was also performed. (Keyword) spoken dialog system / response type selection / encoder-decoder model / multi-task learning (Tokushima University Institutional Repository) ● Metadata: 116668 (Link to Publication Site) ● Publication site (DOI): 10.1016/j.specom.2021.07.003 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1016/j.specom.2021.07.003 (Tokushima University Institutional Repository: 116668, DOI: 10.1016/j.specom.2021.07.003)
11.	ZOLZAYA BYAMBADORJ, Ryota Nishimura, Ayush Altangerel and Norihide Kitaoka : Normalization of Transliterated Mongolian Words Using Seq2Seq Model with Limited Data (accepted), ACM Transactions on Asian and Low-Resource Language Information Processing, Vol.20, No.6, 1-19, 2021. (Summary) The huge increase in social media use in recent years has resulted in new forms of social interaction, changing our daily lives. Due to increasing contact between people from different cultures as a result of globalization, there has also been an increase in the use of the Latin alphabet, and as a result a large amount of transliterated text is being used on social media. In this study, wepropose a variety of character level sequence-to-sequence (seq2seq) models for normalizing noisy, transliterated text written in Latin script into Mongolian Cyrillic script, for scenarios in which there is a limited amount of training data available. We applied performance enhancement methods, which included various beam search strategies, N-gram-based context adoption, edit distance-based correction and dictionary-based checking, in novel ways to two basic seq2seq models. We experimentally evaluated these two basic models as well as fourteen enhanced seq2seq models, and compared their noisy text normalization performance with that of a transliteration model and a conventional statistical machine translation (SMT) model. The proposed seq2seq models improved the robustness of the basic seq2seq models for normalizing out-of-vocabulary (OOV) words, and most of our models achieved higher normalization performance than the conventional method. When using test data during our text normalization experiment,our proposed method which included checking each hypothesis during the inference period achieved the lowest word error rate (WER = 13.41%), which was 4.51% fewer errors than when using the conventional SMT method. (Keyword) Text normalization / noisy text / transliterated text / neural network / language model (Link to Publication Site) ● Publication site (DOI): 10.1145/3464361 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1145/3464361 (DOI: 10.1145/3464361)
12.	JiaHao Chen, Ryota Nishimura and Norihide Kitaoka : End-to-end recognition of streaming Japanese speech using CTC and local attention, APSIPA Transactions on Signal and Information Processing, Vol.9, No.e25, 1-7, 2020. (Summary) Many end-to-end, large vocabulary, continuous speech recognition systems are now able to achieve better speech recognition performance than conventional systems.Most of these approaches are based on bidirectional networks and sequence-to-sequence modeling however, so automatic speech recognition (ASR) systems using such techniques need to wait for an entire segment of voice input to be entered before they can begin processing the data, resulting in a lengthy time-lag, which can be a serious drawback in some applications. An obvious solution to this problem is to develop a speech recognition algorithm capable of processing streaming data. Therefore, in this paper we explore the possibility of a streaming, online, ASR system for Japanese using a model based on unidirectional LSTMs trained using connectionist temporal classification (CTC) criteria, with local attention. Such an approach has not been well investigated for use with Japanese, as most Japanese-language ASR systems employ bidirectional networks. The best result for our proposed system during experimental evaluation was a character error rate of 9.87%. (Keyword) CTC / Local Attention / speech recognition / streaming recognition (Tokushima University Institutional Repository) ● Metadata: 115877 (Link to Publication Site) ● Publication site (DOI): 10.1017/ATSIP.2020.23 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1017/ATSIP.2020.23 (Tokushima University Institutional Repository: 115877, DOI: 10.1017/ATSIP.2020.23)
13.	Norihide Kitaoka, seto eichi and Ryota Nishimura : Example Phrase Adaptation Method for Customized, Example-Based Dialog System Using User Data and Distributed Word Representations, IEICE Transactions on Information and Systems, Vol.E103-D, No.11, 2332-2339, 2020. (Summary) We have developed an adaptation method which allows the customization of example-based dialog systems for individual users by applying plus and minus operations to the distributed representations obtained using the word2vec method. After retrieving user-related profile information from the Web, named entity extraction is applied to the retrieval results. Words with a high term frequency-inverse document frequency (TF-IDF) score are then adopted as user related words. Next, we calculate the similarity between the distrubuted representations of selected user-related words and nouns in the existing example phrases, using word2vec embedding. We then generate phrases adapted to the user by substituting user-related words for highly similar words in the original example phrases. Word2vec also has a special property which allows the arithmetic operations plus and minus to be applied to distributed word representations. By applying these operations to words used in the original phrases, we are able to determine which user-related words can be used to replace the original words. The user-related words are then substituted to create customized example phrases. We evaluated the naturalness of the generated phrases and found that the system could generate natural phrases. (Keyword) example-based spoken dialog system / user customization / distributed word representation (Link to Publication Site) ● Publication site (DOI): 10.1587/transinf.2020EDP7066 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1587/transinf.2020EDP7066 (DOI: 10.1587/transinf.2020EDP7066)
14.	Ryota Nishimura, Daisuke Yamamoto, Takahiro Uchiya and Ichi Takumi : MMDAE: Dialog scenario editor for MMDAgent on the web browser, ICT Express, Vol.5, No.1, 47-51, 2019. (Summary) 我々は，MMDAgent(音声対話システム用の完全オープンソースのツールキット)を開発した．このツールキットは，PCとスマートフォン上の様々なプラットフォーム上で動作する．これに合わせて，対話シナリオの編集環境もさまざまなプラットフォームで動作する必要があるため，Webブラウザで実装されたシナリオエディタを開発する．本論文で開発するシステムの目的は，シナリオを簡単に編集できる環境の提供である．提案されたシナリオエディタを使用して被験者に対して実験を行った．提案システムは，シナリオの読みやすさを向上させ，編集を容易にすることが示された． (Keyword) spoken dialog system / scenario editor / web browser / MMDAgent (Tokushima University Institutional Repository) ● Metadata: 115138 (Link to Publication Site) ● Publication site (DOI): 10.1016/j.icte.2018.03.002 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85046812358 (Tokushima University Institutional Repository: 115138, DOI: 10.1016/j.icte.2018.03.002, Elsevier: Scopus)
15.	Ryota Nishimura, Daisuke Yamamoto, Takahiro Uchiya and Ichi Takumi : Web-based environment for user generation of spoken dialog for virtual assistants, EURASIP Journal on Audio, Speech, and Music Processing, Vol.2018, No.1, 1-13, 2018. (Summary) 本論文では，ユーザがビデオ仮想アシスタントを用いて対話を編集することを可能にするウェブベースの音声対話生成環境を開発した．提案システムでは，誰もが対話システムの対話の内容を簡単に投稿/編集できる．音声対話共有サービスおよびFST生成器は，音声認識装置，対話制御装置，音声シンセサイザ，および仮想エージェントを含むMMDAgent音声対話システムツールキットの音声対話コンテンツを生成する．ダイアログコンテンツの作成には，ユーザーが投稿したQ&AダイアログとFSTテンプレートが使用される．提案されたシステムは，名古屋工業大学の学生ラウンジで1年以上にわたり運用され，実験中に500以上の対話が追加された．最も投稿されたカテゴリーは，「アニメ，ビデオゲーム，漫画」に関するものであった．音声対話システムに関する知識がない観光スタッフによるシステム利用実験を行った結果，スタッフはシステムの応答の長さを短縮し，より長い応答にはポーズを追加することで，ユーザに理解されやすい対話を実現していた． (Keyword) spoken dialog system / MMDAgent / digital signage / real field / web service (Tokushima University Institutional Repository) ● Metadata: 115098 (Link to Publication Site) ● Publication site (DOI): 10.1186/s13636-018-0142-8 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1186/s13636-018-0142-8 (Tokushima University Institutional Repository: 115098, DOI: 10.1186/s13636-018-0142-8)
16.	Yuki Todo, Ryota Nishimura, Kazumasa Yamamoto and Seiichi Nakagawa : Chat-like Spoken Dialogue System with Multiple Agents, The Transactions of the Institute of Electronics, Information and Communication Engineers D, Vol.J99-D, No.2, 188-200, 2016. (Summary) 音声対話システムとの対話において，現在は，ユーザ一人とシステムエージェント一体による二者対話(1 対 1 の対話)が主流である．我々はこれまでに 1 対 1 の雑談向け音声対話システムの開発を行ってきた．このシステムを応用し，本論文ではシステム側のエージェントを二体にした三者対話システムを提案し，開発・評価を行った．雑談に用いた対話ドメインは，「うどんとラーメンのどちらが好きか」といったような，だれでも話しやすい内容にした．そして，同じドメインで対話が可能な二者対話システム(1 ユーザ対 1 システム) と三者対話システム(1 ユーザ対 2 システム)を構築し，これらを用いて被験者実験を行った．この結果から，二者対話システムで得られた知見を，ほとんどそのまま用いた三者対話システムにおいて，対話の弾み具合の向上や，対話の雑談らしさの印象の向上が見られ，被験者に好ましい影響を与えることが明らかになった． (Keyword) 音声対話システム / 多人数対話 / 三者対話 / 複数エージェント / 雑談対話 (Link to Publication Site) ● Publication site (DOI): 10.14923/transinfj.2015JDP7010 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.14923/transinfj.2015JDP7010 (DOI: 10.14923/transinfj.2015JDP7010)
17.	Takahiro Uchiya, Masaki Yoshida, Daisuke Yamamoto, Ryota Nishimura and Ichi Takumi : Design and Implementation of Open-Campus Event System with Voice Interaction Agent, Journal of Mobile Multimedia, Vol.11, No.1, 237-250, 2015. (Summary) 多くの大学では，高校生を募集するためのキャンパス情報を提供するイベントとして積極的にオープンキャンパス活動を行っている．しかし，オープンキャンパス活動にはいくつかの問題がある．私たちは，スマートフォンで動作する音声対話ツールキットを使ってオープンキャンパスイベントを開催するシステムを提案する．この方法は，パンフレットに掲載された情報をブラウザに表示することによって印刷コストを削減することができる．さらに，3Dキャラクターモデル，GPS，Googleマップ，歩行経路推定システムを用いて表現された音声対話エージェントを用いてコースを誘導することにより，人間の案内人は必要ない．これらの機能を利用したオープンキャンパスのイベントシステムの設計と実装を行い，提案システムを実験的に評価した． (Keyword) Open-campus event system / Agent-based campus guide / voice interaction agent (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-84949762177 (Elsevier: Scopus)
18.	Ryota Nishimura and Seiichi Nakagawa : A Spoken Dialog System for Spontaneous Conversations Considering Response Timing and Response Type, IEEJ Transactions on Electrical and Electronic Engineering, Vol.6, No.Supplement 1, S17-S26, 2011. (Summary) 音声対話システムが人間のように自然に応答することができれば，対話はよりスムーズになる．本研究では，人間の行動を模倣する音声対話システムの開発を目指す．提案したシステムは，適切なタイミングで応答を生成するために決定木を利用する．これらの応答には，「相槌」(バックチャネル)，「復唱」，「共同補完」などが含まれる．決定木は，ピッチとパワーの概形，音声認識の途中結果，応答の準備情報を入力として応答タイミング機能を生成します．このモデルの主観評価では，応答内容やタイミングには高い自然性があり，音声対話システムはユーザフレンドリな応答ができることが示された． (Keyword) spoken dialog system / speech communication / aizuchi / repetition / overlapping (Link to Publication Site) ● Publication site (DOI): 10.1002/tee.20616 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1002/tee.20616 (DOI: 10.1002/tee.20616)
19.	Ryota Nishimura, Norihide Kitaoka and 中川聖一 : Analysis of Factors to Make Prosodic Change in Spoken Dialog(<Feature Articles>Rhythm and Timing), Journal of the Phonetic Society of Japan, Vol.13, No.3, 66-84, 2009. (Summary) In smooth and cooperative human-to-human conversations, the prosody as pitch is synchronized between speakers. From this, it is expected that there are some relations between various prosodic changes and the impressions of the conversations. It is necessary to understand what factors are involved in meaning the conversation smooth and lively. In this paper, we analyzed the correlation between the fundamental frequency's synchrony tendency or overlap frequency, and subjective measures of "liveliness" and "familiarity" in human-to-human dialog. As a result, when the prosodic change of speakers synchronized well, it seemed the dialog became familiar, lively, and frank, and the speakers were in agreement. It is shown that the impression signifiers of the conversation are able to be evaluated with only the use of prosodic information. (Keyword) 音声対話 / 韻律同調 / 韻律変化モデル / 応答タイミング / 音声対話システム (Link to Publication Site) ● Publication site (DOI): 10.24467/onseikenkyu.13.3_66 (Link to Search Site for Scientific Articles) ● CiNii @ National Institute of Informatics (CRID): 1390282679763876224 ● Search Scopus @ Elsevier (DOI): 10.24467/onseikenkyu.13.3_66 (DOI: 10.24467/onseikenkyu.13.3_66, CiNii: 1390282679763876224)
20.	Norihide Kitaoka, Masashi Takeuchi, Ryota Nishimura and Seiichi Nakagawa : Response timing detection using prosodic and linguistic information for human-friendly spoken dialog systems, Transactions of the Japanese Society for Artificial Intelligence, Vol.20, No.3, 220-228, 2005. (Summary) If a dialog system can respond to the user as reasonably as a human, the interaction will become smoother. Timing of the response such as back-channels and turn-taking plays an important role in such a smooth dialog as in human-human interaction. We developed a response timing generator for such a dialog system. This generator uses a decision tree to detect the timing based on the features coming from some prosodic and linguistic information. The timing generator decides the action of the system at every 100 ms during the users pause. In this paper, we describe a robust spoken dialog system using the timing generator. Subjective evaluation proved that almost all of the subjects experienced a friendly feeling from the system. (Link to Publication Site) ● Publication site (DOI): 10.1527/tjsai.20.220 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-18544374120 (DOI: 10.1527/tjsai.20.220, Elsevier: Scopus)

Academic Letter:

1.	Ryota Nishimura, Takumi Nagao, Ayato Ichimanda and Norihide Kitaoka : Study on Editing Method to Improve Speech Intelligibility Based on Speech Perception Characteristics of Elderly People, Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.30, No.6, 840-845, 2018. (Summary) 近年の超高齢社会において，高齢者の加齢に伴う聴覚機能の低下が問題視されている．聴覚機能が低下すると，会話によるコミュニケーションにも支障をきたす．近年は，音声対話システムを利用した高齢者サポートシステムなどの開発も行われているが，このようなシステムで用いられる合成音声は高齢者にとっては聞き取りにくいものである．そこで本研究では，まず高齢者に音声の聴取実験を行ってもらい，その結果から聴覚特性の分析を行った．聴取実験では，単語了解度試験を行い，音素単位(子音部，母音部)での識別率を求めた．その結果，摩擦音，破擦音，破裂音同士での異聴が多かった．この結果を受けて，我々は音声に対して高齢者の聴覚特性に基づいた子音強調加工を施し，高齢者にとって聞き取りやすい音声を作成することができるかを調査した．音声加工は，特に異聴が多かった /k/，/s/，/t/，/h/，/ky/，/sy/，/ch/ の音素に対して行った．具体的な加工法としては，子音部の振幅を原音声比 400% で増幅させるものである．加工音声の評価実験では，単語了解度試験による聴取実験を行った被験者と同じ被験者に聴取実験を行ってもらい，得られた聞き取り結果の正答率の比較を行った．結果，いくつかの音素においては，音声を加工することで正答率が上昇した． (Keyword) speech perception characteristics of elderly people / phoneme emphasis processing / amplitude amplification (Link to Publication Site) ● Publication site (DOI): 10.3156/jsoft.30.6_840 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.3156/jsoft.30.6_840 (DOI: 10.3156/jsoft.30.6_840)
2.	Ryota Nishimura, 檜垣美帆 and Norihide Kitaoka : RNN-LSTMによる音響ベクトル空間と文書ベクトル空間とのマッピング, Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.30, No.4, 628-633, 2018. (Summary) 音楽の音響データと歌詞のデータをニューラルネットワークによって対応付ける．音響データをクロマベクトル系列にしてLSTMでベクトルに変換する．その変換先が歌詞ベクトル(Word2vecによる歌詞中の単語のベクトル化したものの平均)になるように学習し，対応付け可能であることを示した． (Link to Publication Site) ● Publication site (DOI): 10.3156/jsoft.30.4_628 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.3156/jsoft.30.4_628 (DOI: 10.3156/jsoft.30.4_628)
3.	Norihide Kitaoka, Shuhei Segawa, Ryota Nishimura and Kazuya Takeda : Recognizing emotions from speech using a physical model, Acoustical Science and Technology, Vol.39, No.2, 167-170, 2018. (Summary) 音声に含まれる特徴を用いて感情を自動推定する手法を検討した．感情としては，ラッセルの2軸のモデルを用い，その2軸を推定することとした．特徴抽出に，音声の母音から声帯の物理モデルのパラメータを推定してそのパラメータを用いることで，従来の特徴よりも推定精度が向上することを示した． (Link to Publication Site) ● Publication site (DOI): 10.1250/ast.39.167 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85043521603 (DOI: 10.1250/ast.39.167, Elsevier: Scopus)

Review, Commentary:

1.	Norihide Kitaoka, Ryota Nishimura and 太田健吾 : Development of a multimodal dialog system with photorealistic CG agent, The Journal of the Acoustical Society of Japan, Vol.78, No.5, 257-264, May 2022. (Summary) When considering the future human-machine cooperative society, an important issue is how natural and easy it will be for machines and humans to communicate with each other.We believe that one way to solve this problem is to have machine interfaces that look as close as possible to humans and interact with them in a manner similar to humans. Therefore, we have focused on "Saya," a realistic 3D computer graphics model that is indistinguishable from a real human and operates realistically in video images, and are constructing a system that enables voice and multimodal dialogue using Saya as an agent. In this paper, we introduce Saya and the overall image of the dialogue system we have built, and outline the speech recognition, speech synthesis, image recognition, and interaction control technologies used in the system. (Link to Publication Site) ● Publication site (DOI): 10.20697/jasj.78.5_257 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.20697/jasj.78.5_257 (DOI: 10.20697/jasj.78.5_257)
2.	大須賀晋, 田中五大, 鍋倉彩那, 藤井宏行, 中野涼太, 渡邉凌太, TELYUKA (名), 太田健吾, Ryota Nishimura and Norihide Kitaoka : Multimodal Agent Development for Locomotion in Next Era, Journal of Society of Automotive Engineers of Japan, Vol.75, No.9, 108-109, Sep. 2021.

Proceeding of International Conference:

1.	Kei Daizumoto, Naoka Osafune, Kohei Torii, Ryota Nishimura, Hisanori Uehara, Mitsuki Nishiyama, Saki Kobayashi, Yutaro Sasaki, Ryotaro Tomida, Yoshito Kusuhara, Tomoya Fukawa, Kunihisa Yamaguchi and Masayuki Takahashi : Deep learning-based depth prediction system for upper tract urothelial carcinoma, the 111th Annual Meeting of the Japanese Urological Association (JUA), PDA-36-05, Yokohama, Apr. 2024. (Keyword) medical image / deep learning
2.	Kei Daizumoto, Naoka Osafune, Kohei Torii, Ryota Nishimura, Hisanori Uehara, Mitsuki Nishiyama, Saki Kobayashi, Yutaro Sasaki, Ryotaro Tomida, Yoshiteru Ueno, Yoshito Kusuhara, Tomoya Fukawa, Kunihisa Yamaguchi, Yasuyo Yamamoto, Masayuki Takahashi and Junya Furukawa : Development of pT classification prediction system in UTUC using deep-learning, 39th Annual European Association of Urology Congress, A0130, Paris, Apr. 2024.
3.	Koharu Horii, Kengo Ohta, Ryota Nishimura, Atsunori Ogawa and Norihide Kitaoka : Language modeling for spontaneous speech recognition based on disfluency labeling and generation of disfluent text, 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 1851-1856, Taipei, Oct. 2023. (Link to Publication Site) ● Publication site (DOI): 10.1109/APSIPAASC58517.2023.10317137 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/APSIPAASC58517.2023.10317137 (DOI: 10.1109/APSIPAASC58517.2023.10317137)
4.	Aito Nakata, Ryota Nishimura, Kengo Ohta and Norihide Kitaoka : Development of a model for predicting timing of back-channel in a real-time spoken dialog system, 2023 IEEE 12th Global Conference on Consumer Electronics, GCCE 2023, 454-457, Nara, Oct. 2023. (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE59613.2023.10315537 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE59613.2023.10315537 (DOI: 10.1109/GCCE59613.2023.10315537)
5.	Shohei Noguchi, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : Extraction of keywords from disease-fighting blogs by fine-tuning the T5 model, 2022 6th International Conference on Natural Language Processing and Information Retrieval, Dec. 2022. (Summary) To measure lifestyle-related diseases such as heart disease and diabetes, which are on the increase, information on patients' lifestyles is necessary. With the growth of the Internet culture in recent years, a great deal of valuable information about patients' long fight against the disease, including lifestyle and condition changes, is now being written in the form of blogs and tweets. In this study, keywords related to changes in health status are extracted from "disease-fighting blogs" using Named Entity Recognition system, which is fine-tuned the "T5 Model", one of the latest deep learning and natural language processing models. This method can be applied to a system that informs patients and health care professionals how their lifestylehabits affect their health status. (Keyword) disease-fighting blog / named entity recognition / T5
6.	Junya Furutani, XIN KANG, Keita Kiuchi, Ryota Nishimura, Manabu Sasayama and Kazuyuki Matsumoto : Learning a Bimodal Emotion Recognition System Based on Small Amount of Speech Data, 2022 8th International Conference on Systems and Informatics (ICSAI), 160-164, Kunming, China, Dec. 2022. (Summary) This paper presents a bimodal emotion recognition system based on the voice and text information using small amount of speech data. Specifically, speech is divided into voice and text to learn emotion classifiers for each modal. The probabilities obtained from these emotion classifiers are weighted based on Mehrabians rule and summed up for each emotion to calculate the final score for bimodal emotion recognition. To create a highly accurate system while solving the problem that there are few Japanese speech data with emotion labels, we propose a novel data augmentation method and employ a transfer learning approach based on a pre-trained VGG16 model and a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model on the tweets. In order to prove the effectiveness of the proposed method, we revealed the recognition results for 7 emotional states, i.e., anger, sadness, joy, fear, surprise, disgust, and neutral. The experiment result suggested that our novel data augmentation method improved the accuracy and that bimodal predictions based on voice and text-based outperformed the single-model predictions. (Link to Publication Site) ● Publication site (DOI): 10.1109/ICSAI57119.2022.10005454 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85146960025 (DOI: 10.1109/ICSAI57119.2022.10005454, Elsevier: Scopus)
7.	Tomoki Kusunose, XIN KANG, Keita Kiuchi, Ryota Nishimura, Manabu Sasayama and Kazuyuki Matsumoto : Facial Expression Emotion Recognition Based on Transfer Learning and Generative Model, 2022 8th International Conference on Systems and Informatics (ICSAI), 107-112, Kunming, China, Dec. 2022. (Summary) Facial expression emotion recognition has been a popular research topic, which played an important role in assisting the natural human-machine conversation. The conventional method for emotion estimation from facial expressions is to learn a CNN-based image classification model from scratch, However, learning such model requires a large number of labeled facial expression images, which is still a limited resource until now. To solve this problem, we propose a data augmentation method based on StyleGAN2 to generate artificial expression images with respect to seven emotions and use them as the additional training data. We further train an expression emotion recognition model based on a VGG16 network through transfer learning. In this research, we proposed a method using transfer learning and augmented images of facial expressions using trained VGG16 and StyleGAN2 and conducted experiments to achieve higher recognition accuracy for racial expression emotion recognition. Our experiment based on the CFEE dataset suggested that an emotion recognition accuracy of 75.10% could be obtained through transfer learning and the accuracy could further improved to 82.04% with the augmented expression images. (Link to Publication Site) ● Publication site (DOI): 10.1109/ICSAI57119.2022.10005478 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85146960739 (DOI: 10.1109/ICSAI57119.2022.10005478, Elsevier: Scopus)
8.	Meiko Fukuda, Masakazu Sugiyama, Ryota Nishimura and Norihide Kitaoka : A Corpus-based Analysis of Age-related Changes in the Acoustic Features of Elderly to Super Elderly Speech, The O-COCOSDA 2022, Hanoi, Vietnam, Nov. 2022. (Summary) Speech recognition technology is a convenient modality for elderly people with limited dexterity and eyesight since it allows them to operate smartphones and other devices, but speech recognition accuracy declines as people get older. Our corpus of elderly Japanese speech, called EARS (Elderly Adults Read Speech), includes many speakers over 80 years old, and we have used it to create acoustic models which improve recognition accuracy for elderly speech [1]. Although the acoustic features of elderly peoples speech differs from those adults [2, 3], the studies compared the acoustic features of the speech of elderly and super elderly Japanese people are not so many. In this paper, we investigated age-related changes in fundamental frequency (Fo), formant frequencies, laryngeal noise, MFCCs and vowel centralization in elderly to super elderly speakers (60-98 years old), using speech data from the EARS corpus and the S-JNAS corpus of elderly Japanese speech. We found that the speech of both genders exhibited an increase in F1 and decrease in F2 of the vowel/i, some similarities in mean and standard deviation in MFCCs, and a decrease in the mean and standard deviation of shimmer with increasing age. In contrast, different aging-related dynamics were observed for male and female speakers in fundamental frequency, formant frequencies other than those listed above, vowel centralization, most other MFCCs. (Link to Publication Site) ● Publication site (DOI): 10.1109/O-COCOSDA202257103.2022.9997946 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85146372335 (DOI: 10.1109/O-COCOSDA202257103.2022.9997946, Elsevier: Scopus)
9.	Mori Daiki, Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Implicit language information replacing method in Japanese encoderdecode ASR model, 2022 9th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA), Tokoname, Japan, Sep. 2022. (Summary) Recent years, automatic speech recognition (ASR) tasks often use language models as an adjunct to ASR models. The density ratio approach (DRA) is one of several language model integration methods. It is known that Japanese has a much larger number of characters than the alphabet language, and that there are variations in reading with homonyms and the same characters. It was unclear whether the implicit language information of character-based encoderdecoder ASR model using beam search algorithm is approximated by the external language model. In our experiments, We have applied an DRA to a Japanese encoderdecoder ASR model to reduce character error rate (CER) in cross-domain scenarios. Cross-domain CERs were calculated for the Japanese academic presentation speech (APS) corpus and the Japanese simulated presentation speech (SPS) corpus. This method achieved a relative error reduction of 11.0% and 22.5% with the RNN and Transformer models compared to the shallow fusion. To investigate the applicability of different speaking styles to different domains, We also conducted an experiment to replace the implicit language information inside the CSJ ASR model with Mainichi Shimbun language model. For the JNAS task, the DRA achieved a relative error reduction of 7.3% compared to the shallow fusion method.
10.	Koharu Horii, Meiko Fukuda, Kengo Ohta, Ryota Nishimura, Atsunori Ogawa and Norihide Kitaoka : End-to-End Spontaneous Speech Recognition Using Disfluency Labeling, Proc. of Interspeech 2022, 4108-4112, Incheon, Korea, Sep. 2022. (Summary) Spontaneous speech often contains disfluent acoustic features such as fillers and hesitations, which are major causes of er- rors during automatic speech recognition (ASR). In this paper, we propose a method of disfluency labeling to address this problem. Our proposed method replaces disfluent phenomena in the transcription of speech data used for training with two types of labels, filler (#) and hesitation (@), and trains an end- to-end ASR model using this data, which makes it possible to recognize disfluent acoustic phenomena as recognition targets, like characters. In addition, by removing the disfluency labels that are included in the recognition results, the words that the speaker actually intended to say can be extracted from the dis- fluent speech. The results of our evaluation experiments show that both the character and sentence error rates were reduced for all of the ASR test sets when disfluency labeling was ap- plied, compared to the baseline method. The proposed method also outperformed other methods intended to reduce disfluency- related errors, even when more disfluent, spontaneous dialog speech was used. This study shows that explicit learning of two disfluent features, fillers and hesitations, is effective in sponta- neous speech recognition. (Keyword) end-to-end speech recognition / spontaneous speech / disfluency / filler / hesitation (Link to Publication Site) ● Publication site (DOI): 10.21437/Interspeech.2022-281 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85140071471 (DOI: 10.21437/Interspeech.2022-281, Elsevier: Scopus)
11.	Naomichi Tabuchi, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : Analysis of Lifestyle Habits from Weblogs of Patients with Intractable Diseases Using Deep Learning, Proceedings of International Conference on Electrical, Computer and Energy Technologies (ICECET), Jul. 2022. (Summary) In recent years, patient-centered medicine has become widely accepted. To keep a record of their struggles,patients sometimes publish their diaries (disease blogs) on weblogs to be read by an unspecified number of people. In particular, writing a blog may be helpful in the case of intractable diseases of which there are few cases, and patients often feel isolated because they know few people with the same disease. In this study, we used deep learning technology to analyze the blogs written by patients with intractable diseasesand to determine how they live their daily lives and their feelings. If their lifestyles are understood, the relationship between the physical condition and mental changes of people fighting intractable diseases can be determined, and a system that shows behavioral guidelines for new users fighting similar intractable diseases can be built. (Keyword) lifestyle habits / intractable disease / deep learning (Link to Publication Site) ● Publication site (DOI): 10.1109/ICECET55527.2022.9872887 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85138876019 (DOI: 10.1109/ICECET55527.2022.9872887, Elsevier: Scopus)
12.	Meiko Fukuda, Ryota Nishimura, Maina Umezawa, Kazumasa Yamamoto, Yurie Iribe and Norihide Kitaoka : Elderly Conversational Speech Corpus with Cognitive Impairment Test and Pilot Dementia Detection Experiment Using Acoustic Characteristics of Speech in Japanese Dialects, Proceedings of the Thirteenth Language Resources and Evaluation Conference, 1016-1022, Marseille, France, Jun. 2022. (Summary) There is a need for a simple method of detecting early signs of dementia which is not burdensome to patients, since early diagnosis and treatment can often slow the advance of the disease. Several studies have explored using only the acoustic and linguistic information of conversational speech as diagnostic material, with some success. To accelerate this research, we recorded natural conversations between 128 elderly people living in four different regions of Japan and interviewers, who also administered the Hasegawas Dementia Scale-Revised (HDS-R), a cognitive impairment test. Using our elderly speech corpus and dementia test results, we propose an SVM-based screening method which can detect dementia using the acoustic features of conversational speech even when regional dialects are present. We accomplish this by omitting some acoustic features, to limit the negative effect of differences between dialects. When using our proposed method, a dementia detection accuracy rate of about 91% was achieved for speakers from two regions. When speech from four regions was used in a second experiment, the discrimination rate fell to 76.6%, but this may have been due to using only sentence-level acoustic features in the second experiment, instead of sentence and phoneme-level features as in the previous experiment. This is an on-going research project, and additional investigation is needed to understand differences in the acoustic characteristics of phoneme units in the conversational speech collected from these four regions, to determine whether the removal of formants and other features can improve the dementia detection rate. (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85144420299 (Elsevier: Scopus)
13.	Naomichi Tabuchi, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : Risk Analysis of Developing Lifestyle-related Diseases based on the Content of Social Networking Service Posts, Innovation in Medicine and Healthcare, Part of the Smart Innovation, Systems and Technologies book series, Vol.308, 63-72, Jun. 2022. (Summary) Japans leading causes of death are cancer, cardiovascular disease, and diabetes, which account for approximately 60% of all deaths. All these diseases are closely related to lifestyle habits such as diet, exercise, and sleep and are called lifestyle-related diseases. Lifestyle-related diseases are characterized by a lack of subjective symptoms, making it difficult to notice the deterioration of the disease. In this study, we analyzed tweets related to diet, exercise, physical condition, and mental state from the contents posted by users on social networking services (SNS) to construct a system that can prevent lifestyle-related diseases more simply. We found that the results of this work can lead to the construction of a system than can detect risk factors leading to the development of lifestyle-related diseases, from the contents of users posts. (Keyword) risk analysis / social networking service / lifestyhle-related diseases (Link to Publication Site) ● Publication site (DOI): 10.1007/978-981-19-3440-7_6 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85135061563 (DOI: 10.1007/978-981-19-3440-7_6, Elsevier: Scopus)
14.	Horii Koharu, Meiko Fukuda, Ohta Kengo, Ryota Nishimura, Ogawa Atsunori and Norihide Kitaoka : End-to-End Spontaneous Speech Recognition Using Hesitation Labeling, Proceedings, APSIPA Annual Summit and Conference 2021, 1077-1081, Tokyo, Dec. 2021. (Summary) Spontaneous speech often contains hesitations, such as stammering, word substitutions, filler words and repetitions, unlike speech that is produced when reading aloud. These addi- tional utterances create noise for Automatic Speech Recognition (ASR) systems, negatively affecting recognition accuracy. In this study, we propose an End-to-End (E2E) ASR system which can recognize these hesitations, as well as extra syllables, in spontaneous Japanese speech by training the ASR model using labeled data. Hesitation speech is then automatically labeled and ignored during speech recognition. Our experiments confirm that both the Character Error Rate (CER) and the Sentence Error Rate (SER) improved for all of the evaluation data sets in comparison to a baseline ASR method. In addition, when examining the actual recognition results, it was observed that the labels were inserted in the correct positions, suggesting that the model is able to correctly learn the meanings of the labels. Furthermore, by deleting the labeled utterances from there cognition results, we were able to obtain grammatically correct target sentences. (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85126651910 (Elsevier: Scopus)
15.	Mori Daiki, Ohta Kengo, Ryota Nishimura, Ogawa Atsunori and Norihide Kitaoka : Advanced language model fusion method for encoder-decoder model in Japanese speech recognition, Proceedings, APSIPA Annual Summit and Conference 2021, 503-510, Tokyo, Dec. 2021. (Summary) We apply an advanced language model fusion method to the encoder-decoder models of a Japanese automatic speech recognition (ASR) system to reduce character error rates (CERs) in cross-domain scenarios. Our method, which uses a Density Ratio approach based on Bayes theorem, is an extension of the Shallow Fusion method widely used for integrating the scores of speech recognition and language models. Since Japanese has a much larger number of characters than alphabet languages, and since there can be multiple possible meanings and pronunciations of the same kanji character, it is unclear whether the linguistic information inside a character- based end-to-end speech recognition is approximated by the language model. We conducted experiments using two types of encoder-decoder models, an RNN model and a Transformer model, and evaluated our results by calculating the cross-domain CERs of the ASR s text output when using the Japanese Academic Presentation Speech (APS) corpus and the Japanese Simulated Presentation Speech (SPS) corpus. When using the RNN model, our proposed method achieved a 1.2% lower CER in comparison to the Shallow Fusion method. When using the Transformer model, our method also outperformed the Shallow Fusion method, achieving a 0.7% lower CER. (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85126661934 (Elsevier: Scopus)
16.	Zolzaya Byambadorj, Ryota Nishimura, Ayush Altangerel, Kengo Ohta and Norihide Kitaoka : Multi-speaker TTS system for low-resource language using cross-lingual transfer learning and data augmentation, Proceedings, APSIPA Annual Summit and Conference 2021, 849-853, Tokyo, Dec. 2021. (Summary) We propose a multi-speaker text-to-speech (TTS) system for use with low-resource languages when only a very small amount of target language data is available. We investigate the effects of using high-resource language datasets and augmented data during model training, and compare different strategies for fine-tuning the model. After using a combination of cross-lingual transfer learning, a small amount of target language data and augmented target language data for training our low-resource TTS model, we then fine-tuned the pre-trained model using the original and augmented target language data. Our experimental results show that by sequentially training the model with high-resource language data, target language data and augmented target language data, followed by gradual fine-tuning using the original and augmented target language data, our system was able to achieve the most natural speech after text-to-speech conversion, achieving a native speaker mean opinion score of 3.50 on a scale of 0 to 5. (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85126663888 (Elsevier: Scopus)
17.	Ryota Nishimura : IEEE GCCE 2021 Tutorial "Research Life Hacks", IEEE GCCE 2021, Osaka, Oct. 2021.
18.	Meiko Fukuda, Nishizaki Hiromitsu, Iribe Yurie, Ryota Nishimura and Norihide Kitaoka : Improving Speech Recognition for the Elderly: A New Corpus of Elderly Japanese Speech and Investigation of Acoustic Modeling for Speech Recognition, Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 6578-6585, May 2020. (Summary) In an aging society like Japan, a highly accurate speech recognition system is needed for use in electronic devices for the elderly, but this level of accuracy cannot be obtained using conventional speech recognition systems due to the unique features of the speech of elderly people. S-JNAS, a corpus of elderly Japanese speech, is widely used for acoustic modeling in Japan, but the average age of its speakers is 67.6 years old. Since average life expectancy in Japan is now 84.2 years, we are constructing a new speech corpus, which currently consists of the utterances of 221 speakers with an average age of 79.2, collected from four regions of Japan. In addition, we expand on our previous study (Fukuda, 2019) by further investigating the construction of acoustic models suitable for elderly speech. We create new acoustic models and train them using a combination of existing Japanese speech corpora (JNAS, S-JNAS, CSJ), with and without our super-elderly speech data, and conduct speech recognition experiments. Our new acoustic models achieve word error rates (WER) as low as 13.38%, exceeding the results of our previous study in which we used the CSJ acoustic model adapted for elderly speech (17.4% WER). (Keyword) Japanese speech corpus / elderly / acoustic modeling (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85096619800 (Elsevier: Scopus)
19.	Jiahao Chen, Ryota Nishimura and Norihide Kitaoka : E2E Streaming Speech Recognition Using CTC and Local Attention, Proc. NCSP'20, 1-4, Feb. 2020.
20.	Obashi Yuya, Ryota Nishimura and Norihide Kitaoka : Automatic Conversion of Written Language into Spoken Language Using a Sequence-to-Sequence Model Trained with a Parallel Corpus, Proceedings of The 22nd Conference of the Oriental COCOSDA (Oriental-COCOSDA2019), 1-5, Sebu, Oct. 2019. (Summary) In this study we proposed using a sequence-to-sequence,RNN based model to convert Japanese written language intoa text representation of Japanese spoken language. If thisprocess could be accomplished accurately and efficiently, itwould become possible to create a large, spoken languagetext corpus for improving the accuracy of speech recognition.We first created a written language-spoken language parallelcorpus based on the Nagoya University Conversation Corpusand the Transcribed Corpus of Elderly Dialog. Using thismanually constructed parallel corpus, we devised conversionmodel and converted the BCCWJ corpus into spoken languagetext. Although conversion accuracy was not impressive overall,some short sentences were converted accurately. Moreover,even if whole sentences could not be converted accurately,the statistics of the spoken language were well expressed inthe converted sentences. Thus, a language model trained witha corpus of spoken language, created by converting writtenlanguage into spoken language, was shown to be effectivefor speech recognition. Because of an insufficient amount oftraining data, we were only able to accurately convert someof the shorter sentences in the BCCWJ corpus, accounting foronly a small percentage of the data. This prevented the creationof a large corpus of spoken language text data. Therefore,it will be necessary to devise a method which can correctlyconvert longer sentences as well. As a solution to this problem,we plan to introduce an attention mechanism to our sequenceto-sequence model.
21.	Meiko Fukuda, Ryota Nishimura, Nishizaki Hiromitsu, Iribe Yurie and Norihide Kitaoka : A New Corpus of Elderly Japanese Speech for Acoustic Modeling, and a Preliminary Investigation of Dialect Dependent Speech Recognition, Proceedings of The 22nd Conference of the Oriental COCOSDA (Oriental-COCOSDA2019), 1-6, Sebu, Oct. 2019. (Summary) We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. (Link to Publication Site) ● Publication site (DOI): 10.1109/O-COCOSDA46868.2019.9041216 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85083226710 (DOI: 10.1109/O-COCOSDA46868.2019.9041216, Elsevier: Scopus)
22.	Komatsu Akihisa, Ryota Nishimura and Norihide Kitaoka : Environmental Sounds Recognition with Convolutional-LSTM, 2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 717-719, Osaka, Oct. 2019. (Summary) This paper addresses the task of recognizing environmentalsounds using the AudioSet data set. Specifically,features were extracted by spectrogram conversion of AudioSet's10-second sound data, and learning was performed by neuralnetwork with 3 layer CNN and the one layer CLSTM , and acomparison experiment with the LSTM model was performed.As a result, CLSTM increased 4 points from 52% to 56%better recognition accuracy than LSTM model, demonstratingthe usefulness of CLSTM in environmental sound recognitionfield. (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE46687.2019.9015256 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE46687.2019.9015256 (DOI: 10.1109/GCCE46687.2019.9015256)
23.	Yamamoto Taiki, Ryota Nishimura, Misaki Masayuki and Norihide Kitaoka : Small-Footprint MagicWord Detection Method Using Convolutional LSTM Neural Network, Proc INTERSPEECH, Vol.2019-Septe, 2035-2039, Graz, Sep. 2019. (Summary) The number of consumer devices which can be operated byvoice is increasing every year. Magic Word Detection (MWD),the detection of an activation keyword in continuous speech, hasbecome an essential technology for the hands-free operation ofsuch devices. Because MWD systems need to run constantlyin order to detect Magic Words at any time, many studies havefocused on the development of a small-footprint system. In thispaper, we propose a novel, small-footprintMWDmethod whichuses a convolutional Long Short-Term Memory (LSTM) neuralnetwork to capture frequency and time domain features overtime. As a result, the proposed method outperforms the base-line method while reducing the number of parameters by morethan 80%. An experiment on a small-scale device demonstratesthat our model is efficient enough to function in real time. (Keyword) keyword spotting / convolutional neural network / recurrent neural network / convolutional LSTM / small footprint (Link to Publication Site) ● Publication site (DOI): 10.21437/Interspeech.2019-1662 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.21437/Interspeech.2019-1662 (DOI: 10.21437/Interspeech.2019-1662)
24.	Takahiro Uchiya, Ryota Nishimura, Takahiro Hirano and Masaru Sakurai : Design of Reminiscence Therapy System for Elderly People with Dementia, Advances on Broad-Band Wireless Computing, Communication and Applications: Proceedings of the 12th International Conference on Broad-Band Wireless Computing, Communication and Applications (BWCCA-2017), 844-853, Nov. 2018. (Summary) 近年，認知症患者の数が増加し，介護者の負担が増大している．しかし，認知症の治療法は確立されておらず，その進行を制御することが治療の目的となっている．認知症の進行を抑制する1つの手段として，介護者と認知症患者との対話による回想法を使用する．しかし，この方法では，介護者は何度も対話を行う必要があり，介護者にとって負担となるため，介護者の代わりにMMDAgentという音声対話システムを用いて回想法対話の支援を行う方法を提案する．これによって，介護者の負担が軽減される．本稿では，提案されたシステムの設計を示す．
25.	Seto Eichi, Ryota Nishimura and Norihide Kitaoka : Customization of an example-based dialog system with user data and distributed word representations, APSIPA ASC 2018, No.8659459, 1718-1724, Nov. 2018. (Summary) 事例ベースの音声対話システムにおいて，限られた事例を，ユーザ固有の適切な事例に自動変換することにより，音声対話システムをユーザに適応する．ユーザのプロフィール情報からWeb検索によって得られた関連語を事例の単語と置き換える方法でユーザ適応を可能にした． (Link to Publication Site) ● Publication site (DOI): 10.23919/APSIPA.2018.8659459 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85063450866 (DOI: 10.23919/APSIPA.2018.8659459, Elsevier: Scopus)
26.	Kanta Kiyohara, Ryota Nishimura and Norihide Kitaoka : Multi-modal Geometry Tutoring System Using Speech and Touchscreen Figure Tracing, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE2018, 252-256, Nara, Oct. 2018. (Summary) 人に対して数学の幾何問題の解答を説明する際には，一般に指差しで図をなぞったりしながら口頭で解説をする．この指と音声のマルチモーダル入力を受け付けて，幾何問題の解答を画面上に表示する幾何問題教育システムを開発した． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2018.8574714 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85060307292 (DOI: 10.1109/GCCE.2018.8574714, Elsevier: Scopus)
27.	Meiko Fukuda, Ryota Nishimura, Hiromitsu Nishizaki and Norihide Kitaoka : Construction of a Corpus for Elderly Japanese Speech Recognition, 2018 IEEE 7th Global Conference on Consumer Electronics GCCE2018, 259-262, Nara, Oct. 2018. (Summary) 情報弱者となる可能性の高い高齢者にとって，音声インタフェースは重要な役割を果たす可能性がある．高齢者にとって性能の高い音声認識器を作るためには高齢者の音声データを収集することが不可欠である．そこで100名以上の平均年齢80歳を超える高齢者の音声を収集してデータベース化した． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2018.8574795 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85060293760 (DOI: 10.1109/GCCE.2018.8574795, Elsevier: Scopus)
28.	Ryota Nishimura, Miho Higaki and Norihide Kitaoka : Mapping Acoustic Vector Space and Document Vector Space by RNN-LSTM, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE2018, 698-699, Nara, Oct. 2018. (Summary) 音楽の音響データと歌詞のデータをニューラルネットワークによって対応付ける．音響データをクロマベクトル系列にしてLSTMでベクトルに変換する．その変換先が歌詞ベクトル(Word2vecによる歌詞中の単語のベクトル化したものの平均)になるように学習し，対応付け可能であることを示した． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2018.8574867 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85060288222 (DOI: 10.1109/GCCE.2018.8574867, Elsevier: Scopus)
29.	Kajinami Kazuaki, Ryota Nishimura and Norihide Kitaoka : Construction of dialog database for development of spoken dialog breakdown detection methods, ICAICTA-2018, 91-95, Aug. 2018. (Summary) 対話システムにおいて，システムが対話破綻を招く発話をすることを自ら発見すれば対話破綻を回避したりすることができる．音声対話において対話破綻検出器を構築するために，音声対話データベースを構築した． (Link to Publication Site) ● Publication site (DOI): 10.1109/ICAICTA.2018.8541273 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85059958707 (DOI: 10.1109/ICAICTA.2018.8541273, Elsevier: Scopus)
30.	Norihide Kitaoka, Nakagawa Takuma, Ryota Nishimura, Ishiguro Yoshio, Kojima Shin'ichi and Ohsuga Shin : A multimodal control system for autonomous vehicles using speech, gesture, and gaze recognition, DSP in vehicles 2018, 2018. (Summary) 音声，ジェスチャ，視線認識を用いた自動運転車のためのマルチモーダルコントロールシステムであり，実際にシステムを構築し，自動運転車に搭載した．また，システムについては，実験用にタクシーに搭載し，タクシー運転手を自動運転車システムに見立てて，被験者実験を行い，自動運転車制御に必要なマルチモーダル理解についての知見を得た．
31.	Kengo Ohta, Rikito Marumoto, Ryota Nishimura and Norihide Kitaoka : Selecting Type of Response for Chat-like Spoken Dialogue Systems Based on Acoustic Features of User Utterances, Proceedings of APSIPA Annual Summit and Conference 2017, No.December, 1-5, Kuala Lumpur, Dec. 2017. (Summary) 本稿では，ユーザ発話から得られた音響特徴を使用して，相槌応答，話題の変更，または話題の拡張などの音声対話システムにおける応答のタイプを選択する．これらの特徴には，MFCCとLSPによるスペクトル情報，F0で表されるピッチ情報，ラウドネスなどが含まれる．高齢者と面接者の対話コーパスを構築し，評価実験の結果，音声セグメント識別タスクで49.3%の精度を得られた．さらに，各素性のデルタ係数を利用することでさらに結果が改善された． (Link to Publication Site) ● Publication site (DOI): 10.1109/APSIPA.2017.8282230 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85050499738 (DOI: 10.1109/APSIPA.2017.8282230, Elsevier: Scopus)
32.	Ryota Nishimura, Takahiro Uchiya, Takahiro Hirano and Masaru Sakurai : Proposal of Reminiscence Therapy System using Spoken Dialog to Suppress Dementia, 2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 418-419, Nagoya, Oct. 2017. (Summary) 近年，認知症患者の数が増加している．介護者の負担も増えている．しかし，認知症の確立された治療法は存在しない．認知症の進行を抑制することは，認知症治療の重要な目標である．回想法は，認知症の進行を抑制する1つの手段である．回想法では，介護者が認知症患者と話をすることが必要であるが，何回も繰り返し会話を行う必要があり，介護者の負担は大きい．そこで，本研究では，MMDAgentを用いた回想支援システムを提案する． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2017.8229394 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85045726578 (DOI: 10.1109/GCCE.2017.8229394, Elsevier: Scopus)
33.	Takuma Nakagawa, Ryota Nishimura, Yurie Iribe, Yoshio Ishiguro, Shin Ohsuga and Norihide Kitaoka : A Human Machine Interface Framework for Autonomous Vehicle Control, 2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 413-415, Nagoya, Oct. 2017. (Summary) 本研究では，自動運転車とのコミュニケーションのために，音声，ジェスチャー，および視線認識を使用した直感的なマルチモーダルインターフェースシステムを開発する．有限状態トランスデューサを用いて，インタフェースシステムのマルチモーダル理解コンポーネントと対話制御コンポーネントを別々に設計した．実装に関しては，従来の対話システムは，入力としてのユーザ動作と出力としてのシステム動作とを有する有限状態変換器によって制御されているが，我々のマルチモーダル理解部と対話制御コンポーネントは，2つの別々のトランスデューサの重ね合わせとして見ることができる．カスケード変換器は，1つの変換器で構成することができる．我々のシステムは自律型自動車シミュレータでシミュレータ上の車を操作することができる． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2017.8229312 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85045760877 (DOI: 10.1109/GCCE.2017.8229312, Elsevier: Scopus)
34.	Takahiro Uchiya, Satoshi Otake, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Extraction of User Preferences based on Voice Interaction, 2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 416-417, Nagoya, Oct. 2017. (Summary) 名古屋工業大学の研究グループは，音声対話ツールキットとして「MMDAgent」を開発している．システム開発者は，MMDAgentを使用して，さまざまな音声対話コンテンツを作成できる．開発者が音声対話のコンテンツを作成する際には，ユーザーのニーズを考慮することが重要であるので，ユーザの嗜好情報を引き出す手法が必要である．本稿では，音声対話からユーザの嗜好情報を抽出する手法を提案する． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2017.8229370 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-85045757721 (DOI: 10.1109/GCCE.2017.8229370, Elsevier: Scopus)
35.	Takeshi Morita, Yu Sugawara, Ryota Nishimura and Takahira Yamaguchi : Implementing customer reception service in robot cafe using stream reasoning and ROS based on PRINTEPS, CEUR Workshop Proceedings, Vol.1960, 1-4, Kobe, Oct. 2016. (Summary) 我々は，実践知能アプリケーション開発プラットフォームであるPRINTEPSを開発した．本稿では，PRINTEPSに基づいたストリーム推論とロボットオペレーティングシステム(ROS)を用い，知識処理とイメージセンシングを統合することで，ロボットカフェにおける顧客受付サービスのPRINTEPSの応用を紹介する．このプラットフォームに基づいて，我々は，適用可能なルールセットを変更することによって，ロボットカフェ内のロボットの挙動を修正できることを実証する．
36.	Ryota Nishimura, Yutaka Takase and Yukiko Nakano : Development environment of a spoken dialogue system based on PRINTEPS, 2016 IEEE 5th Global Conference on Consumer Electronics, 272-273, Kyoto, Oct. 2016. (Summary) 本稿では，PRINTEPSアーキテクチャに基づく音声対話システムについて述べる．この音声対話システムは，5つのモジュール(音声認識，言語理解，対話管理，応答生成，音声合成)で構成されている． PRINTEPSでは，音声対話システムを呼び出すとき，システム開発者は小規模の対話目標を指定する．システムは，ユーザから必要な情報を得るために，ユーザとの対話を実行する．対話目標に対応する対話処理ルールは予め用意されている． PRINTEPSベースのシステム開発の利点は，システム開発者が音声対話システムを知らなくても音声対話システムを構築できることである． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2016.7800412 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2016.7800412 (DOI: 10.1109/GCCE.2016.7800412)
37.	Ryuhei Kani, Takahiro Uchiya, Daisuke Yamamoto, Ichi Takumi and Ryota Nishimura : Proposal of the communication mechanism among voice-interactive digital signage systems, 2016 IEEE 5th Global Conference on Consumer Electronics, 1-2, Kyoto, Oct. 2016. (Summary) 2011年より，筆者らは名古屋工業大学のユーザーに有用な大学情報を提供するために，スタンドアロンの音声対話型デジタルサイネージシステムを開発し，実際に設置・運用した．このシステムには， "Mei"という名前の3次元キャラクターとの音声対話を実現するソフトウェアツールキットであるMMDAgentが含まれる．欠点として，ネットワーク通信機能がないために，このシステムの能力が限られる．本研究では，いくつかのデジタルサイネージシステム間のネットワーク通信機能を新たに実装した．この機能により，情報共有や他の手段による複数の看板システムの共同作業が可能になる． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2016.7800468 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2016.7800468 (DOI: 10.1109/GCCE.2016.7800468)
38.	Yuto Ishida, Takahiro Uchiya, Kouhei Yamamoto, Daisuke Yamamoto, Ryota Nishimura and Ichi Takumi : Related Word Recommendation Mechanism for Speech Dialogue System, 2016 19th International Conference on Network-Based Information Systems (NBiS), 464-468, Ostrava, Czech Republic, Sep. 2016. (Summary) 近年，音声対話システムが著しく発展している．我々の大学では， "MMDAgent"音声対話システムツールキットをオープンソースソフトウェアとして開発し公開している． MMDAgentは，音声認識と音声合成に "Julius"と "HTS"を使用する．そのエージェントは，これらを3D CGレンダリングモジュールと組み合わせて構成する． MMDAgentアプリケーションの一例として，「スマートメイちゃん」が開発され，スマートフォンに移植されている．「スマートメイちゃん」は，音声対話システムとして非常に高性能であるとはいえ，ユーザーの視点から考えると未解決の難しさが残っている．初心者ユーザは何を話すのか分からない．魅力的な会話を誘導するために，「スマートメイちゃん」に関連する単語推薦機構を開発する．それによって，初心者のユーザーには，良い会話ができるトピックが提示される．我々は．会話のための推薦語を用いたシステムの有効性を検証した． (Link to Publication Site) ● Publication site (DOI): 10.1109/NBiS.2016.53 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/NBiS.2016.53 (DOI: 10.1109/NBiS.2016.53)
39.	Takeshi Morita, Yu Sugawara, Ryota Nishimura and Takahira Yamaguchi : Integrating Symbols and Signals Based on Stream Reasoning and ROS, Knowledge Management and Acquisition for Intelligent Systems, Vol.9806, 251-260, Phuket, Thailand, Aug. 2016. (Summary) 実践的知能アプリケーション開発プラットフォームであるPRINTEPS(PRactical INTElligent aPplicationS)を開発した．本稿では，ストリーム推論とROS(Robot Operating System)を用いたイベント検出のためのPRINTEPSの応用と，知識処理による画像センシングの統合について述べる．このプラットフォームにおいて，我々はロボットカフェ内のロボットの動作をデモンストレーションし，ルールセットを変更することによって簡単にロボットカフェを実現・適用できることを示す． (Keyword) ROS / Stream Reasoning / PRINTEPS / Ontology
40.	Takahiro Uchiya, Ryosuke Nakano, Daisuke Yamamoto, Ryota Nishimura and Ichi Takumi : Extension with Intelligent Agents for the Spoken Dialogue System for Smartphones, 2015 IEEE 4th Global Conference on Consumer Electronics (GCCE), 281-282, Osaka, Oct. 2015. (Summary) 我々は，音声対話システム開発ツールキットであるMMDAgentを開発・公開している．このシステムで開発できる音声対話システムをスマートフォンに対応させ，スマートフォン上で音声対話シナリオを構築する際に，複雑なシナリオを構築することは難しいので，本研究ではこの問題を解決するために，スマートフォン用音声対話システムをソフトウェアエージェントによって拡張することを提案する． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2015.7398666 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2015.7398666 (DOI: 10.1109/GCCE.2015.7398666)
41.	Yuichi Matsushita, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Experiment and Evaluation of Crowdsourcing Model for Creation of Voice Interaction Scenario, 2015 IEEE 4th Global Conference on Consumer Electronics (GCCE), 304-305, Osaka, Oct. 2015. (Summary) 本研究では，我々が公開している音声対話システムツールキットであるMMDAgentを使用して音声対話システムを構築する場合の，音声対話シナリオの構築方法を提案します．本研究では，クラウドソーシングを用いて，音声対話を構築・収集するために，音声対話システムをインターネットを介して配布する方法や，対話シナリオを編集するためのwebアプリケーションのテスト動作の結果を示す．提案システムを用いることにより，音声対話コンテンツ作成環境を簡単に構築できる． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2015.7398647 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2015.7398647 (DOI: 10.1109/GCCE.2015.7398647)
42.	Yuichi Matsushita, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Crowdsourcing Environment to Create Voice Interaction Scenario of Spoken Dialogue System, 2015 18th International Conference on Network-Based Information Systems (NBiS), 499-504, Taipei, Sep. 2015. (Summary) 本研究では，我々が公開している音声対話システムツールキットであるMMDAgentを使用して音声対話システムを構築する場合の，音声対話シナリオの構築方法を提案します．本研究では，クラウドソーシングを用いて，音声対話を構築・収集するために，音声対話システムをインターネットを介して配布する方法や，対話シナリオを編集するためのwebアプリケーションのテスト動作の結果を示す．提案システムを用いることにより，音声対話コンテンツ作成環境を簡単に構築できる． (Link to Publication Site) ● Publication site (DOI): 10.1109/NBiS.2015.74 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/NBiS.2015.74 (DOI: 10.1109/NBiS.2015.74)
43.	Ryota Nishimura, Takahiro Uchiya, Daisuke Yamamoto and Ichi Takumi : Development of a dialogue scenario editor on a web browser for a spoken dialogue system, the second international conference on Human-agent interaction (HAI '14), 129-132, Tsukuba, Oct. 2014. (Summary) 我々は，音声対話システムの対話シナリオを作成する際の環境の改善を図るために，音声対話シナリオエディタを開発した．我々が開発・公開している音声対話システム構築ツールキットであるMMDAgentは，様々なプラットフォーム・OS上で動作するが，これに合わせて，シナリオエディタも様々な環境で動作させる必要がある．そこで，我々は，音声対話シナリオエディタをウェブブラウザ上に実装した．このシステムを用いて被験者実験を行ったところ，対話シナリオ編集時の可読性が向上し，シナリオ編集のしやすさも向上した． (Link to Publication Site) ● Publication site (DOI): 10.1145/2658861.2658904 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1145/2658861.2658904 (DOI: 10.1145/2658861.2658904)
44.	Daisuke Yamamoto, Keiichiro Oura, Ryota Nishimura, Takahiro Uchiya, Akinobu Lee, Ichi Takumi and Tokuda Keiichi : Voice interaction system with 3D-CG virtual agent for stand-alone smartphones, the second international conference on Human-agent interaction (HAI '14), 323-330, Tsukuba, Oct. 2014. (Summary) 本稿では，スタンドアロンのスマートフォンに3D-CGバーチャルエージェントを用いた音声対話システムを提案する．提案システムは既存のモバイル用音声対話システムとは異なり，スタンドアロンのスマートフォンで音声認識や音声合成を行うことができるため，遅延なく自然に会話することができる．さらに，提案システムは，対話スクリプト，Javaベースのプラグイン，およびAndroid APIによってカスタマイズすることができる．したがって，音声対話システム開発者は提案されたシステムに基づいて簡単にスマートフォン用の元の音声対話システムを作成することができる．我々は，提案システムのサブセットをオープンソースソフトウェアとして利用可能にした．このシステムは，スマートフォンを使用した人間とエージェントの対話の研究に貢献することが期待される． (Link to Publication Site) ● Publication site (DOI): 10.1145/2658861.2658874 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1145/2658861.2658874 (DOI: 10.1145/2658861.2658874)
45.	Ryota Nishimura, Koji Sugioka, Daisuke Yamamoto, Takahiro Uchiya and Ichi Takumi : A VoIP-based voice interaction system for a virtual telephone operator using video calls, 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), 529-532, Tokyo, Oct. 2014. (Summary) 我々は，電話オペレータシステム音声対話システムを提案する． Microsoft LyncをVoIPクライアントとして使用して，ビデオ通話を行う．音声対話システムは，3Dキャラクタを表示し，ビデオ通話を通じてユーザと対話するためにシステムで使用される．この研究では，電話オペレータのインタラクティブなシナリオが大学の従業員データベースから自動的に生成される．このシステムを使用した被験者からの実験結果は，良好なシステム性能を示した． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2014.7031317 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2014.7031317 (DOI: 10.1109/GCCE.2014.7031317)
46.	Yuichi Matsushita, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Construction of crowdsourcing environment for creation of voice interaction scenario, 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), 689-690, Tokyo, Oct. 2014. (Summary) 本研究では，我々が公開している音声対話システムツールキットであるMMDAgentを使用して音声対話システムを構築する場合の，音声対話シナリオの構築方法を提案します．本研究では，クラウドソーシングを用いて，音声対話を構築・収集するために，音声対話システムをインターネットを介して配布する方法や，対話シナリオを編集するためのwebアプリケーションのテスト動作の結果を示す．提案システムを用いることにより，音声対話コンテンツ作成環境を簡単に構築できる． (Link to Publication Site) ● Publication site (DOI): 10.1109/GCCE.2014.7031156 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/GCCE.2014.7031156 (DOI: 10.1109/GCCE.2014.7031156)
47.	Takahiro Uchiya, Daisuke Yamamoto, Ryota Nishimura, Ichi Takumi and Tetsuo Kinoshita : Proposal of voice control-based interactive design method of agent system, 2014 IEEE 13th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), 496-500, London, Aug. 2014. (Summary) エージェントシステムは近年のエージェント技術を用いてデザイン・実装されている．しかし，これらのデザインや実装はまだ困難である．エージェントシステムのデザイン・開発をスムーズに行うために，我々は，音声コントロールベースの対話デザイン手法を提案し，これをエージェントリポジトリベースのマルチエージェントフレームワークに適用します． (Link to Publication Site) ● Publication site (DOI): 10.1109/ICCI-CC.2014.6921505 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/ICCI-CC.2014.6921505 (DOI: 10.1109/ICCI-CC.2014.6921505)
48.	Yuki Todo, Ryota Nishimura, Kazumasa Yamamoto and Seiichi Nakagawa : Development and evaluation of spoken dialog systems with one or two agents through two domains, Text, Speech, and Dialogue, Vol.8082 LNAI, 185-192, Berlin Heidelberg, Sep. 2013. (Summary) Almost all current spoken dialog systems treat dialog as that where a single user talks to an agent. On the other hand, we set out to investigate a multiparty dialog system that deals with two agents and a single user. We developed a three person (one user and two agents) and a two person (one user and one agent) dialog system to consider the same dialog tasks, that is, Which do you prefer, udon or ramen (Japanese noodle or Chinese noodle)? and Which do you want to travel to Hokkaido or Okinawa (snowy region or tropical region)? and compared them with respect to user behavior and satisfaction. According to the results of the experiments, the three person dialog system performed better in terms of lively conversation, and user can talk with the agents more like chatting. (Keyword) spoken dialog system, / multi-party dialogue / two agents / chat (Link to Publication Site) ● Publication site (DOI): 10.1007/978-3-642-40585-3_24 (Link to Search Site for Scientific Articles) ● Summary page in Scopus @ Elsevier: 2-s2.0-84884966427 (DOI: 10.1007/978-3-642-40585-3_24, Elsevier: Scopus)
49.	Yuki Todo, Ryota Nishimura, Kazumasa Yamamoto and Seiichi Nakagawa : Development and evaluation of spoken dialog systems with one or two agents, 14th Annual Conference of the International Speech Communication Association (INTERSPEECH-2013), 1896-1900-, Lyon, Aug. 2013. (Summary) 現存する音声対話システムのほとんどは，ユーザとシステムが一対一の対話である．我々は，多人数対話に興味があり，本研究では，エージェント側を一人または二人にすることで，最大3人対話を行うことができる音声対話システムを開発した．このシステムを用いて，被験者実験を行い，三者対話によって対話感が向上した．
50.	Ryota Nishimura, Yuki Todo, Kazumasa Yamamoto and Seiichi Nakagawa : Chat-like Spoken Dialog System for a Multi-party Dialog Incorporating Two Agents and a User, The 1st International Conference on Human-Agent Interaction (iHAI 2013), Vol.II-p13, 1-8, Sapporo, Aug. 2013. (Summary) 現存する音声対話システムは，一対一対話のものがほとんどであるが，我々は，ユーザ一人に対してシステムが二人存在する，三者対話を行う音声対話システムを開発した．このシステムは，ユーザとシステムの二者対話を行うシステムを簡単に拡張したものである．被験者実験の結果，三者対話システムはより良い対話感をユーザに与え，より雑談らしく対話がおこなえるという評価を得た．
51.	Ryota Nishimura and Seiichi Nakagawa : Response timing generation and response type selection for a spontaneous spoken dialog system, 2009 IEEE Workshop on Automatic Speech Recognition Understanding, 462-467, Merano, Italy, Dec. 2009. (Summary) もし音声対話システムが人間と同じように自然な応答を返すことができれば，対話をより自然に行うことができる．本研究では，我々は，雑談中の人間の対話行動を模倣する音声対話システムを開発した．このシステムでは，応答の種類として相槌，復唱，共同補完などを行うことができる．また応答タイミングは，オーバーラップやバージインなどを行うことが可能である．被験者実験を行った結果，本システムを用いることで，被験者は音声対話システムに対して親しみを感じていることが示された． (Link to Publication Site) ● Publication site (DOI): 10.1109/ASRU.2009.5372898 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.1109/ASRU.2009.5372898 (DOI: 10.1109/ASRU.2009.5372898)
52.	Toshihiko Itoh, Norihide Kitaoka and Ryota Nishimura : Subjective experiments on influence of response timing in spoken dialogues, Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2009, 1835-1838, Brighton, Sep. 2009. (Summary) これまでの研究からの対話リズムに関する分析結果の妥当性を検証するため，応答タイミングや他の対話に関する分析結果をもとに対話を作成し，対話の自然性，対話の自然性，合成音声，および発声の理解の容易さを含を検証するために被験者実験を行った．実験の結果，発話には自然な応答タイミングが存在し，発話内容に応じて適切な応答タイミングが存在していることが示された．
53.	Ryota Nishimura, Norihide Kitaoka and Seiichi Nakagawa : Analysis of relationship between impression of human-to-human conversations and prosodic change and its modeling, 9th Annual Conference of the International Speech Communication Association (INTERSPEECH-2008), 534-537, Brisbane, Sep. 2008. (Summary) もし音声対話システムが人間同士の対話のように自然に応答を返すことができれば，対話はよりスムースになる．人間が対話中に行う韻律制御を音声対話システムに実現させるために，我々は人間同士の対話中に起こる韻律変化と，その対話の印象との関係を分析した．また，この分析結果を元に，韻律制御モデルを構築し，音声対話システムに実装した．
54.	Ryota Nishimura, Norihide Kitaoka and Seiichi Nakagawa : A spoken dialog system for chat-like conversations considering response timing, Text, Speech, and Dialogue, 599-606, Sep. 2007. (Summary) If a dialog system can respond to a user as naturally as a human, the interaction will be smoother. In this research, we aim to develop a dialog system by emulating the human behavior in a chat-like dialog. In this paper, we developed a dialog system which could generate chat-like responses and their timing using a decision tree. The system could perform collaborative completion, aizuchi (back-channel) and so on. The decision tree utilized the pitch and the power contours of users utterance, recognition hypothesis, and response preparation status of the response generator, at every time segment as features to generate response timing.
55.	Ryota Nishimura, Norihide Kitaoka and Seiichi Nakagawa : Prosody change and response timing analysis in spontaneously spoken dialogs and their modeling in a spoken dialog system, 8th Annual Conference of the International Speech Communication Association (INTERSPEECH-2007), 2565-2568, Antwerp, Belgium, Aug. 2007. (Summary) 対話システムが人間のように自然な応答ができれば，対話はよりスムーズになる．人間の音声のの韻律的な振る舞いを模倣することは，コンピュータと人間との自然な会話の実現おいて重要である．本論文では，協調的で友好的な音声対話システムを開発するために，人間同士の対話においてF0の同期傾向や「盛り上がり」，「親しみ」などの主観的指標について分析した．また，これらの特徴をモデル化し，相槌の応答タイミングを生成するモデルを構築し音声対話システムに実装した．

Proceeding of Domestic Conference:

1.	DATE Ryuto, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Evaluation of Speech Recognition with Lip Information Using Deep Learning in Various Noise Types, 日本音響学会研究発表会講演論文集, No.1-Q-34, 1-4, Sep. 2024. (Summary) In recent years, the development of speech recognition technology has improved the accuracy of speech recognition using deep learning models, and systems based on these models have become popular. However, in real-world situations, noise is often present in speech signals, resulting in a significant degradation of recognition accuracy. Therefore, how to improve noise immunity is an issue in the development of practical systems. Multimodal speech recognition is one way to improve noise reduction. In humans, information is transmitted through the auditory, visual, and tactile senses, which are referred to as modalities. The state in which information is transmitted by using these modalities differently or simultaneously is called a multimodal state [1]. In noisy environments, speech information alone is susceptible to noise and may reduce recognition accuracy, but visual information such as lip information is unaffected by noise. In this study, experiments were conducted using the NTCD-TIMIT dataset[2] . The phone error rate (PER) was used for evaluation, and the recognition rates were compared under various noise conditions. We also evaluated which phonemes were better recognized when lip information was used, compared to when only speech information was used.
2.	金井良賢, 若林佑幸, Ryota Nishimura and Norihide Kitaoka : Estimation of the end time of input utterances to a spoken dialogue system considering linguistic features using wav2vec 2.0, 日本音響学会研究発表会講演論文集, No.2-Q-50, 1-2, Sep. 2024. (Summary) In this study, we proposed a method for estimating in advance the end time of the other party's speech in order to enable the system to smoothly change speakers. Experimental results showed that MFCC and wav2vec 2.0 could be used to estimate the end time of the other party's speech. The results also suggest that the prediction accuracy may be improved by lengthening the input speech duration. As a future prospect, we would like to conduct further experiments using linguistic features, as we believe that contextual information and the content of the conversation are effective in predicting the end time of speech. Although we were able to confirm that the accuracy varied with the length of the input speech, we were not able to obtain enough significance to determine the details of the trend.
3.	SUGIYAMA Masakazu, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Turn-taking prediction based on acoustic features, 日本音響学会研究発表会講演論文集, No.2-Q-48, 1-4, Sep. 2024. (Summary) We proposed a turn-taking prediction model that uses only acoustic features for faster decision making. By using only acoustic features, the waiting time for speech recognition is eliminated and the time from the end of speech to the start of judgment is shortened. Experiments were conducted using several model configurations, including a change in the pre-training model and multi-task learning, but the accuracy rate was about 75% for all configurations. In the future, we would like to review the length of speech input to the model and the model configuration to achieve faster turn-taking prediction and to improve the accuracy of the prediction.
4.	HAYAMICHI Hirotaka, Kazuyuki Matsumoto, Ryota Nishimura and Minoru Yoshida : Improvement of health maintenance support system using a conversation robot by adding chat functions, Proceedings of the 38th Annual Conference of the Japanese Society for Artificial Intelligence, 1-4, May 2024. (Summary) We are developing a system that helps people improve their lifestyles by using an interactive robot and a web application that analyzes the content of the dialogue and suggests lifestyle habits. Using this system, we actually collected and analyzed lifestyle habits and evaluated the system, analyzing problems in the functionality and UI. Based on the results of this analysis, we aim to improve the system to make it more effective. This paper mainly describes the addition of a chat dialogue function and improvements to the analysis of user responses. The aim is to arouse the users interest in the system through the use of chat dialogues, to encourage continuous use of the system, and to motivate the user to improve his or her lifestyle. In addition, we confirmed that the use of GPT-3.5 in the analysis of user responses improves the accuracy of the Yes/No judgment of user responses. (Keyword) communication robot / health management system / chat dialogue / speech recognition
5.	TAKANABE Toshiki, Kazuyuki Matsumoto, 木内敬太, XIN KANG, Ryota Nishimura and 篠山学 : 感情分析のためのカウンセリングマルチモーダルデータセットの構築および評価, 情報処理学会全国大会講演論文集, 4-83-4-84, Mar. 2024. (Summary) コロナ禍をきっかけとして，オンラインカウンセリングが急激に普及した．オンラインカウンセリングは，細かな表情や声色の変化の認識が難しいことが問題点として挙げられる．カウンセラーの判断を助け，業務効率化のために，クライアントの感情の自動分析が有効である．人間は相手の感情を読み取るとき，声色，表情，発話内容など複数の要素から総合的に判断を行っている．AIによる感情推定も同様に，複数の異なるモダリティを組み合わせて用いるマルチモーダル感情推定により，高精度な感情分析が可能になると考えられる．マルチモーダル感情推定には，学習用データとして，感情のラベルが付与されたマルチモーダルデータセットが必要となる．本研究では，オンラインカウンセリング中のクライアントの感情分析のためのマルチモーダルデータセットを構築し，その評価を行う．本データセットには，カウンセラーによる労務者を対象としたオンラインカウンセリングにおける労務者の様子を収録した動画と，第三者により付与された客観感情ラベルが含まれている． (Keyword) マルチモーダル感情分析 / カウンセリング面談 / フライスのカッパ係数 / ラッセルの円環モデル / オンラインカウンセリング
6.	FUKUNAGA Yoshinori, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Construction of a Backchannel Surface Selection Model for Natural Spoken Dialog Systems Us- ing Deep Learning, 日本音響学会研究発表会講演論文集, No.2-P-4, 949-952, Mar. 2024. (Summary) In this paper, we propose a deep learning model that uses BERT to select an appropriate verbal accompaniment from among 11 types of accompaniments based on the input text from the point where the accompaniment occurs up to 4 utterances before the occurrence of the accompaniment. Although the apparent accuracy of the proposed method on the test data was low (about 30%), the actual use of the model resulted in natural azuragi. When multiple responses were considered as correct responses, the accuracy of the model was 81.2% in terms of F value. Since there are multiple natural responses to an utterance, it is possible to improve the accuracy of the model by preparing and learning multiple correct answer labels. In the future, we plan to switch to a multi-label classification method to improve the aizuchi selection model.
7.	EMOTO Jotaro, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Development of Noise and Silence Rejection Real-time VAD-less Speech Recognition Model, 日本音響学会研究発表会講演論文集, No.2-P-6, 957-960, Mar. 2024. (Summary) In this study, we developed a speech recognition system that can input speech containing noise and silence as it is, using a deep learning model, Wav2Vec 2.0. We trained a model that can determine noise/silence intervals by superimposing noise on the speech in the training data and tagging the corresponding intervals. In order to verify whether the proposed model is able to determine noise/silence intervals outside of speech segments, we conducted recognition experiments using multiple noise/silence lengths and achieved a rejection rate of approximately 100% for the determination of silence intervals and approximately 90% for the determination of noise intervals. In addition, a comparison of the speech recognition accuracy using CER between the models with and without tagging during training showed a relative improvement of about 19 % in CER, and an improvement in accuracy for two types of errors, i.e., substitution errors and deletion errors. In this study, we used XLSR-53 from the publicly available Wav2Vec 2.0 pre-training models, but there are several other models available that differ in the number of speech data used in the pre-training and in the number of parameters of the model. By comparing these models, we would like to investigate models that are capable of recognizing speech segments outside of the speech segment.
8.	IMAMOTO Rintaro, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Constructing and Evaluating a Real-time Spoken Dialog System Incorporating Models of Back- channel Responses and Turn-Taking, 日本音響学会研究発表会講演論文集, No.2-P-6, 957-960, Mar. 2024. (Summary) Recently, spoken dialogue systems have become familiar to us due to the spread of voice assistants. However, these systems do not respond to the user while the user is speaking, but wait for a while after the user finishes speaking before responding. On the other hand, in human-to-human dialogues, the listener responds to the speaker without a long pause after the speaker finishes his/her utterance, and the system does not respond to the listener. In order to realize such a human-to-human dialogue between a system and a user, it is necessary for the user to respond with appropriate timing. In this study, as a method for constructing a spoken dialogue system that returns appropriate responses at the appropriate timing, we used a deep learning model that can rapidly predict the timing of appropriate responses using only acoustic features. We evaluated the timing of the system's responses through experiments with subjects. The subjects were asked to compare the proposed system with the baseline system that uses the speech recognition results and acoustic features.
9.	Meiko Fukuda, Ryota Nishimura, Iribe Yurie, Yamamoto Kazumasa and Norihide Kitaoka : EARSConstruction of the speech corpus of Japanese Super-rderlies, 日本音響学会研究発表会講演論文集, No.1-2-4, 763-766, Mar. 2024. (Summary) We are currently building a corpus of Japanese elderly-as-a-service (EARS) to improve the accuracy of speech recognition for elderly-as-a-service (S-JNAS). This corpus is positioned as a very elderly version of S-JNAS. At the present stage, EARS has a database of 123 speakers' speech, which is available to the public. Although the total recording time is only 13.4 hours, speech recognition experiments using ESPnet 2 have shown an error rate of 8.3% for very elderly people's voices. This is 3.5% lower than the error rate when EARS is not used for training.
10.	金井良賢, 若林佑幸, Ryota Nishimura and Norihide Kitaoka : Prior estimation of utterance end time for smooth spoken dialogue systems, 日本音響学会研究発表会講演論文集, No.2-P-7, 961-964, Mar. 2024. (Summary) In this study, we proposed a method for estimating in advance the end time of the other party's speech in order to enable the system to smoothly change speakers. Experimental results showed that MFCC and wav2vec 2.0 could be used to estimate the end time of the other party's speech. The results also suggest that the prediction accuracy may be improved by lengthening the input speech duration. As a future prospect, we would like to conduct further experiments using linguistic features, as we believe that contextual information and the content of the conversation are effective in predicting the end time of speech. Although we were able to confirm that the accuracy varied with the length of the input speech, we were not able to obtain enough significance to determine the details of the trend.
11.	KOTO Kaito, Ryota Nishimura, Ohta Kengo and Norihide Kitaoka : Construction and evaluation of a multilingual speech synthesis model with monolingual speakers, 日本音響学会研究発表会講演論文集, No.2-Q-37, 1267-1270, Sep. 2023.
12.	Tsuyoshi Honda, Kohei Torii, Ryota Nishimura, Minoru Yoshida and Kazuyuki Matsumoto : Segmentation of Dental Panoramic X-ray Images Using the Deep Learning Model TransUNet, 第22回情報科学技術フォーラム (FIT2023), 571-572, Sep. 2023. (Summary) 歯科パノラマ X 線画像は歯の診療に使われる画像である．歯科パノラマの照合には専門的な知識が必要であり，歯科医師の負担軽減を目的として歯の自動診断システムの研究が行われてきた．その中で，深層学習技術を用いて歯の識別を行う状態分類モデルが開発されている．本研究では状態分類モデルの入力として必要な歯の輪郭情報を作成するため，深層学習技術を用いて歯科パノラマ X 線画像のセグメンテーションを行う．実験では，歯科クリニックで撮影，ラベル付けされたデータを取り扱う．セグメンテーションモデルには，近年医療セグメンテーションタスクで成功を収めているTransUNet モデルを用いて実験を行った． (Keyword) パノラマX線画像 / セグメンテーション / deep learning / TransUNet
13.	RYOTAROH Nagae, Kohei Torii, Ryota Nishimura, Kazuyuki Matsumoto and Minoru Yoshida : Automatic Classification Method for Dental Implants Using Deep Segmentation Model, 第22回情報科学技術フォーラム (FIT2023), 567-568, Sep. 2023. (Summary) 歯科インプラントとは，失った歯を補うための医療機器である．インプラントは，メーカーや種類が様々あり，それぞれで治療法が異なるため，歯科医師は治療時にインプラントの種類を把握しておく必要がある．本研究では，歯科医師の負担軽減を目的として，患者の歯科デンタル X 線画像から深層学習を用いたセグメンテーションモデルによってインプラントの領域を切り取り，画像分類モデルを用いてインプラントの分類を行う．切り抜きの精度は，IoU と Dice 係数でそれぞれ，0.923，0.857 となった．U-Net によってインプラントの領域を切り抜いた画像に対する分類の評価は，正解率，適合率，再現率，F 値がそれぞれ，0.851，0.852，0.884，0.840 となった． (Keyword) デンタルX線画像 / 画像認識 / deep learning
14.	Kenta Kumakura, Kohei Torii, Ryota Nishimura, Kazuyuki Matsumoto and Minoru Yoshida : Diagnosis of Teeth Condition in Dental Panoramic X-Ray Images and Validation of Generalization Performance, 第22回情報科学技術フォーラム (FIT2023), 563-564, Sep. 2023. (Summary) 近年, 医療ビッグデータの深層学習技術への活用が盛んに行われており, 医療ビッグデータから診療情報や画像, カルテ情報等を用いて診断結果の提示や診断援助等ができるようになってきた. 本研究では, 歯科医師の負担軽減と診断援助を目的として, 深層ニューラルネットワークアーキテクチャを利用した歯科パノラマ X 線画像における歯の状態診断を行う. 実験では, 共同研究先の歯科クリニックで撮影, ラベル付けされたデータを取り扱う. 歯は複数の状態を持つため, 分類モデルはマルチラベル分類手法を用いて実装する. 結果として, う蝕や根尖病巣の識別が困難であった. また, 未知の撮影機材由来のデータでは精度の低下が見受けられた. そのため, ドメイン汎化手法を用いて, さらなる検証を行った. (Keyword) パノラマX線画像 / 画像認識 / deep learning
15.	SUGIYAMA Masakazu, Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Real-time turn-taking system that can handle barge-in utterances, 日本音響学会研究発表会講演論文集, No.2-3P-1, 847-850, Mar. 2023. (Summary) With the recent development of speech recognition technology, spoken dialogue systems have become capable of accurate conversation. However, most existing dialogue systems such as Siri and Google assistant are one-question-one-answer dialogue systems, and user speech is ignored while the system is speaking, so natural conversation is not possible. To solve this problem, the system needs to respond to interrupted speech from the user. Existing spoken dialogue systems that support interrupted speech often stop system speech in response to user interrupts and perform speaker changeover. However, user interjections also include those that do not require speaker changeover, such as aisatsuji. Therefore, in this study, we propose a deep learning model that judges whether the system should change speakers in response to user speech. The proposed model judges whether a user's utterance is a conversation with a word, a laugh, or something else, based only on the first 300 ms of the user's utterance. The proposed model can make a speaker replacement decision without waiting for the end of the user utterance.
16.	Horii Koharu, Ohta Kengo, Ryota Nishimura, Ogawa Atsunori and Norihide Kitaoka : Language Modeling Based on Disfluent Sentence Generation with BERT for Spontaneous Speech Recognition, 日本音響学会研究発表会講演論文集, No.1-3-2, 661-664, Mar. 2023. (Summary) In this study, BERT (Bidirectional Encoder Representations from Transformers) [12], a transformer encoder model that can account for bidirectional context, was used to insert fillers or slurring into the written word. One of the tasks that BERT can solve is to predict and fill in words that fill in blanks in a sentence with blanks. One of the tasks that BERT can solve is to predict and fill in words that fill in the blanks of a sentence with a non-fluent label. In this study, BERT was finetuned with a Masked Language Modeling (MLM) task using transcriptions of labeled spontaneous utterances. MLM is a self-supervised learning task in which BERT masks sentences by itself and learns bidirectional expressions by predicting masked tokens. We expect that MLM can take the context into account and insert non-fluent labels at appropriate positions in the sentence.
17.	DATE Ryuto, Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Improving Speech Recognition Accuracy in Noisy Situations Using Deep Learning with Lip Infor- mation, 日本音響学会研究発表会講演論文集, No.1-3P-3, 777-780, Mar. 2023. (Summary) In recent years, the development of speech recognition technology has improved the accuracy of speech recognition using deep learning models, and systems based on these models have become popular. However, in real-world situations, noise is often introduced into the speech signal, resulting in a significant degradation of recognition accuracy. Therefore, how to improve noise immunity is an issue in the development of practical systems. Multimodal speech recognition is one way to improve noise reduction. In humans, information is transmitted through the auditory, visual, and tactile senses, which are referred to as modalities. The state in which information is transmitted by using these modalities differently or simultaneously is called a multimodal state [1]. Even if there is a possibility of misrecognition with a single modality, the redundancy of multimodality makes it possible to correct or supplement the misrecognition. In this study, multimodal speech recognition was performed using speech and lip information. Since the phonemes that are likely to be misrecognized differ between speech and lip information, we expect to improve the recognition accuracy by using both of them together.
18.	Manabu Sasayama, Keita Kiuchi, XIN KANG, Ryota Nishimura and Kazuyuki Matsumoto : インタビュー対話における問い返し文の生成に関する検討, 言語処理学会第29回年次大会発表論文集, 405-407, Mar. 2023. (Summary) 遠隔勤務をしているとき，遠隔労働者が自分自身のストレスに気づけないことも多い．精神面の健康状態が深刻になる前に自分自身で気づくことができれば，健康を維持することにつながる．本研究では，遠隔労働者が健康状態をセルフチェックするためのバーチャルエージェント対話システムの構築を目的としている．本稿では，対話時に必要となる「問い返し」の生成手法について考察する．
19.	Kazuyuki Matsumoto, 木内敬太, XIN KANG, Ryota Nishimura and 篠山学 : Relationships Between Client Stress and Multimodal Emotion Estimation in Counseling Situations, JSKE 第18回日本感性工学会春季大会講演論文集, Mar. 2023. (Summary) This study aims to develop a technology to detect the degree of stress of remote workers, which has been increasing in recent years, from multimodal data such as video and audio data during interviews. In this paper, we use interview data obtained in preliminary experiments to analyze the relationship between emotions estimated from facial, audio, and verbal information and stress evaluation by interview. Using the emotion estimation results for each modal based on machine learning algorithm as input, we train a model to predict stress evaluation results, evaluate errors, and confirm the validity of stress evaluation using the emotion estimation results. In addition, by clustering the time-series change patterns of the emotion estimation results, we will analyze whether or not there is a relationship between each type of emotion and stress evaluation. (Keyword) automatic stress detection / multimodal emotion estimation / remote worker
20.	Hirotaka Hayamichi, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : Development of health maintenance support system using spoken dialogue robots, HAI シンポジウム2023 講演論文集, Mar. 2023. (Summary) Lifestyle-related diseases account for 30% of all medical costs and 60% of all deaths. It is important to maintain good lifestyle habits to protect one's health. In recent years, an increasing number of services and devices allow voice input. Speech recognition makes it easier to use computers. In our study, we develop a system that helps people improve their lifestyle habits by using a web application that analyzes the content of conversations with an interactive robot and suggests lifestyle habits and other information. Using the developed system, we actually collected lifestyle habits, analyzed them, and evaluated the system. (Keyword) health maintenance support system / spoken dialogue robots / lifestyle habits
21.	ISHIHARA Reo, Kei Daizumoto, 堀克仁, Fumiya Kadoriku, Yutaro Sasaki, TOMIDA Ryotaro, Yoshiteru Ueno, Megumi Tsuda, Yoshito Kusuhara, Tomoya Fukawa, Yasuyo Yamamoto, Kunihisa Yamaguchi, Masayuki Takahashi, Hiro-omi Kanayama and Ryota Nishimura : Deep-learningを用いた尿管癌についての検討, 第266会徳島医学会学術集会(令和4年度冬期), No.A-25, 266, Feb. 2023. (Summary) Although diagnostic imaging techniques have evolved in recent years, the accuracy of ureteral cancer detection and depth is still low. In particular, ureteral cancer depth diagnosis technology is important before drug therapy, and its early development is required. In this study, we conducted a pilot study to construct a system to predict the depth of ureteral cancer from CT images using AI (artificial intelligence) based on deep learning (deep learning). Methods: We evaluated the learning rate of the system by classifying CT images of 30 cases of lower ureteral cancer out of 157 cases of ureteral cancer treated by total nephroureteral resection from 2009 to 2020 at the University of Tokushima Hospital. The location of cancer was determined by comparing the location of cancer in the macro image of the resected specimen with the location of cancer in the CT image, and each CT slice was classified as "containing cancer" or "not containing cancer". The correct rate of lower ureteral cancer determination in CT images by the AI was 99.94% when tested with trained data at the time of 100 training cycles, and 99.15% when tested with un-trained data. No over-learning was observed in this study. [Conclusion] We evaluated lower ureteral cancer in CT images using AI. It was considered possible to learn to determine the presence or absence of cancer from CT images in this setting. In the future, we will use AI to identify the location and predict the depth of ureteral cancer from CT images, and aim to establish an evaluation model that can be applied in actual clinical practice.
22.	大須賀晋, 田中五大, 鍋倉彩那, 中野涼太, 渡邊凌太, 石川友香, 石川晃之, 中村晃一, 藤井裕也, 堀内颯太, 東中竜一郎, Ryota Nishimura, 太田健吾 and Norihide Kitaoka : EMOtive A.I. ``Saya'', 人工知能学会研究会資料言語・音声理解と対話処理研究会, Vol.96, 33, Dec. 2022. (Summary) EMOtive A.I. is an autonomous dialogue system that combines the latest image recognition, voice recognition, speech synthesis, dialogue AI, action and drawing control based on psychological knowledge, and character expression by CG artists. EMOtive A.I. is an autonomous dialogue system that combines the latest image recognition, speech recognition, speech synthesis, and dialogue A.I., as well as behavior and drawing control based on psychological knowledge and character expression by CG artists. Following the interview and the real-time dialogue demonstration with Kaiyodo Senmu at the Wonder Festival 2022 summer main stage, the demo system will be exhibited at the Shibuya Ward event SIW2022 in November, where it interacted with any general user at the Seibu Shibuya store. (Link to Publication Site) ● Publication site (DOI): 10.11517/jsaislud.96.0_33 (Link to Search Site for Scientific Articles) ● Search Scopus @ Elsevier (DOI): 10.11517/jsaislud.96.0_33 (DOI: 10.11517/jsaislud.96.0_33)
23.	akihiro Torii, Ryota Nishimura and Norihide Kitaoka : Construction of a Dialogue Breakdown Detector for Spoken Dialogue Systems, 令和4年度電気・電子・情報関係学会四国支部連合大会講演論文集, 145, Sep. 2022. (Summary) However, it is often the case that the system responds inappropriately to the user's utterances, resulting in the breakdown of dialogue. In recent years, research has been conducted to solve this problem, focusing on methods for detecting dialogue breakdowns based on linguistic information, with particular attention paid to chat-based dialogue systems. On the other hand, there are very few examples of research on speech-based dialogue systems, and further progress is desired.In this paper, we construct a detector model composed of neural networks and aim at estimating dialogue breakdowns based on acoustic features, which are information unique to spoken dialogue.
24.	Kohyoh Fukumura, Ryota Nishimura and Norihide Kitaoka : Chat Dialogue Topic Extension by BERT, 令和4年度電気・電子・情報関係学会四国支部連合大会講演論文集, 146, Sep. 2022. (Summary) Dialogue systems require many response sentences when a person converses with a system, but preparing many response sentences requires a lot of manpower and is costly. In this study, we propose a method for automatically expanding the topics of response sentences by using text data available on the Internet, a unique expression extractor, and BERT[1].
25.	堀井こはる, Meiko Fukuda, 太田健吾, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : Evaluation of End-to-End Disfluency Shaping Speech Recognition System Using Spoken Dialogue, 日本音響学会講論集, Vol.2-8-5, 1155-1156, Sep. 2022. (Summary) Automatic Speech Recognition (ASR) is now widely used, has been extensively studied, and continues to improve significantly in accuracy. In particular, end-to-end (E2E) models, in which the mapping from input to output is learned in an integrated manner, are the mainstream ASR models these days [1]. Our everyday speech contains fillers such as "uh," "um," etc., which are one of the causes of misrecognition in speech recognition [2]. They are also unnecessary when using transcribed data. In order to deal with such nonfluent acoustic features, we replaced the filler and slurring parts of transcribed Japanese spontaneous speech with nonfluent labels, which we call symbols (filler "#", slurring "@"), and trained the ASR model as a single recognition target like a character in E2E. We constructed an "end-to-end non-fluent formatted speech recognition system," which obtains recognition results from non-fluent speech by shaping the non-fluent parts of the speech. We conducted evaluation experiments using conference lectures and mock lectures, and showed improvement in recognition accuracy [3, 4]. In this paper, we present an evaluation experiment using dialogue speech to demonstrate the effectiveness of the system for spontaneous speech of another domain.
26.	森大輝, Kengo Ohta, Ryota Nishimura and Norihide Kitaoka : Design of Encoder-Decoder speech recognition models augmented with out-of-domain acoustic information, 日本音響学会講論集, Vol.2-Q-26, 1229-1232, Sep. 2022. (Summary) While various methods have been studied to improve recognition accuracy by augmenting the language information inside a speech recognition model with an external language model, there has been little research on augmenting the acoustic information in a speech recognition model. Therefore, we applied the idea of DRA to reinforce the acoustic information inside the speech recognition model with out-of-domain acoustic information. In the proposed method, the out-of-domain acoustic information extracted by subtracting the output probability of the out-of-domain language model from that of the out-of-domain speech recognition model is added to the target-domain speech recognition model.In our experiments, we used the trans- former model and RNN model as speech recognition models to verify the proposed method. Using the Japanese academic lecture speech recognition model and the Japanese simulated lecture speech recognition model, we calculated CERs and cross-domain CERs for each domain and compared them with the baseline and Shallow Fusion.
27.	Meiko Fukuda, masakazu sugiyama, Ryota Nishimura, Yurie Iribe, Kazumasa Yamamoto and Norihide Kitaoka : Acoustic characteristics of the elderlies in the super-elderly speech corpus EARS and S-JNAS, 日本音響学会講論集, Vol.3-Q-32, 1395-1398, Sep. 2022. (Summary) Speech recognition is a convenient modality for elderly people who have limited fingers and eyesight to operate smartphones and other devices, but it has been reported that the accuracy of speech recognition decreases with age [1, 19]. Therefore, we have developed a large corpus of elderly adults' speech (EARS: Elderly Adults Read Speech), especially those over 80 years old, and used it to create an acoustic model to improve the recognition accuracy [2]. Although elderly people's speech has various acoustic features compared to those of adolescents and middle-aged people [5, 6, 7], there are not many reports on very elderly people. In this paper, we investigated the age-related changes of fundamental frequency (Fo), formant frequency, laryngeal noise, MFCC, and vowel centralization in the EARS corpus and the S-JNAS corpus of elderly speakers aged 60-98 years.
28.	Naomichi Tabuchi, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura and Kenji Kita : SNS の投稿内容に基づく生活習慣病発症リスクの分析, 言語処理学会第28回年次大会発表論文集, 1946-1950, Mar. 2022. (Summary) 国内における死亡原因は，がん・循環器疾患・糖尿病などで全体の約 6 割を占めている．これらの病気はいずれも食事・運動・睡眠などの生活習慣と深い関りがあり，生活習慣病と呼ばれている．生活習慣病には自覚症状がほとんどないため，病気の悪化に気づきにくいという特徴がある．本研究では，より簡易的に生活習慣病を予防するシステムの構築を目指し，ソーシャル・ネットワーキング・サービス(SNS)を用いて，ユーザの投稿内容から食事・運動・体調・精神状態に関するツイートの分析を行った．調査の結果，本研究の成果がユーザの投稿内容から生活習慣病発症の危険因子を検出するシステムの構築につながることが分かった． (Keyword) Social Networking Services / lifestyle-related disease / SNS / BERT / Latent Dirichlet Allocation
29.	森大輝, 太田健吾, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : Designing an end-to-end speech recognition model with off-target acoustic information, 日本音響学会講演論文集, Vol.2-3Q-2, 1141-1144, Mar. 2022. (Summary) 従来の音声認識器では DNN-HMM 音声認識モデルが使用されていたが，音響，辞書および言語モデルなどの様々なモジュールで構成されている非常に複雑なシステムであった.一方で End-to-End 音声認識は複雑な処理を単一のニューラルネットワークを用いて表現可能であるため，従来の音声認識システムを大幅に簡素化することが可能となった．特に Encoder- Decoder モデルは音声認識や機械翻訳を含む自然言語処理タスクにおいて，認識精度を著しく改善している. Encoder-Decoder 音声認識モデルは, 十分な量の学習データがあれば, 高精度なモデルを作成することが可能だが, 大量のテキストデータで学習した言語モデルを併用することで, 豊富な言語情報を活用でき, さらに高い性能を得ることができる．音声認識モデルと言語モデルの統合方法として, Shallow Fusion[1] と呼ばれる手法がある．他にも, Deep Fusion や Cold Fusion も提案されている．このように, 音声認識モデルと言語モデルの両方を利用する様々なアプローチが存在する．近年, Shallow Fusion を確率的に改善した Density Ratio Approach[2] と呼ばれる手法が提案されており, この手法は音声認識モデル内部で学習される言語情報である ``暗黙の言語情報'' を除去した時の出力確率にターゲットドメイン言語モデルの出力確率を加算することにより, 音声認識モデルの言語情報を差し替えることが可能である．Density Ratio Approach に対して様々な研究が行われており, 既に英語, スペイン語, イタリア語, 日本語において有効であることが証明されている．[3, 4, 5] 音声認識モデルの認識精度を言語モデルによる文脈情報の増強で改善する手法が様々検討されている一方で, 音声認識モデルの音響情報を増強する研究はあまり行われていない．そこで我々は Density Ratio Approach の考え方を応用し, ドメイン外の音響情報を付加するための手法を提案する．この手法では, ドメイン外音声認識モデルの出力確率から, ドメイン外言語モデルの出力確率を減算することで抽出したタスク外音響情報をターゲットドメイン音声認識モデルに加算する．我々の実験では, 音声認識モデルとして, Trans- former モデルを使用した．日本語学術講演音声認識モデルと日本語模擬講演音声認識モデルを用いて, それぞれのドメインに対する CER を算出した．いずれの実験においても, Shallow Fusion および Baseline の認識精度を改善することはできなかった．
30.	堀井こはる, Meiko Fukuda, 太田健吾, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : End-to-End Speech Recognition with Hesitation Identification and Removal Based on Disfluency Labeling, 日本音響学会講論集, Vol.1-3-5, 889-892, Mar. 2022. (Summary) The accuracy of Automatic Speech Recognition (ASR) continues to improve significantly with the introduction of end-to-end (E2E) models [1], but non-fluent acoustic features such as fillers are one of the main causes of misrecognition in speech recognition for spoken words.In speech recognition for spoken language, nonfluent acoustic features such as fillers are one of the main causes of misrecognition.To address this problem, we have developed a number of methods, including one that directly models the problem by using a Hidden Markov Model (HMM) or an N-gram language model [2], one that integrates a hidden event model with an acoustic or language model [3], one that uses a garbage model [4], and one that uses an acoustic or language model [5]. Garbage model [4], and other HMM-based methods have been proposed.These methods require capturing a wide range of acoustics within the structure of the HMM, making it difficult to design models.There are also methods [5][6] that identify faltering words and fillers by acoustic features and remove them before recognition, but in these approaches, the accuracy of detection by acoustic features may affect the accuracy of speech recognition.Lou et al. [7] attempted to have a recognizer produce fluent translations directly from nonfluent speech, and they used Connectionist Tempo- ral Classification (CTC), Long Short-Term Memory (LSTM), and E2E ASR. Memory (LSTM), or Transformer-based ASR models could produce fluent transcriptions without explicitly detecting non-fluent parts. In [8], the RNN-Transducer model explicitly labels and recognizes the non-fluent parts of speech as non-fluent parts of speech.In this study, in spontaneous speech in Japanese, we replaced the filler and the slurring parts of the transcriptions with non-fluent labels, and used them as symbols (in this study, slurring "@" and filler "#"), and recognized them as one recognition target like characters in the Joint CTC-Attention [9] Transformer The ASR model is trained with E2E to obtain recognition results from non-fluent speech by shaping the slurring.
31.	森大輝, Kengo Ohta, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : Designing an end-to-end speech recognition model with off-target acoustic information, 日本音響学会講論集, Vol.2-3Q-2, 1141-1144, Mar. 2022. (Summary) While various methods have been studied to improve the recognition accuracy of speech recognition models by enhancing contextual information using language models, there has been little research on enhancing acoustic information in speech recognition models.Therefore, we propose a method to add acoustic information outside the domain by applying the Density Ratio Approach.In this method, acoustic information outside the task domain, which is extracted by subtracting the output probability of the language model outside the domain from the output probability of the speech recognition model outside the domain, is added to the target domain speech recognition model.In our experiments, we used the trans- former model as the speech recognition model.We calculated CERs for each domain using the Japanese academic lecture speech recognition model and the Japanese simulated lecture speech recognition model.In both experiments, we could not improve the recognition accuracy of Shallow Fusion and Baseline.
32.	Ryota Nishimura : 自然な対話を可能とする音声対話システムの実現に向けた研究, IEEE札幌支部合同学術講演会, Nov. 2021.
33.	堀井こはる, Meiko Fukuda, 太田健吾, Ryota Nishimura and Norihide Kitaoka : 言い淀みを考慮したEnd-to-End音声認識, 第19回情報学ワークショップ(WiNF2021), Vol.S-5-2, Nov. 2021.
34.	森大輝, 太田健吾, Ryota Nishimura and Norihide Kitaoka : Encoder-Decoder音声認識モデルにおける暗黙的言語情報の置換法, 第19回情報学ワークショップ(WiNF2021), Vol.S-5-3, Nov. 2021.
35.	Tsubasa Miyashita, Minoru Yoshida, Kazuyuki Matsumoto, Ryota Nishimura and Kenji Kita : Lifestyle Extraction from Corpus of Disease Fighting Blogs, Proceedings of 2021 Shikoku-Section Joint Convention Record of the Institutes of Electrical, Electronic, and Information Related Engineers, 187, Sep. 2021. (Summary) 闘病ブログは，患者の生活に関する行動情報(病状・食事・運動)について書かれているものがあり，疾患への理解や納得を得られるものも見られ，同病者の励みとなっている．闘病ブログコーパスとは，闘病ブログを収集し，記事のタグ付けを行ったものである．本研究では年々増加傾向にある2 型糖尿病患者による闘病ブログコーパスから，闘病活動に関わる生活習慣の分析を行う．この分析結果より，病状悪化の危険因子や病状改善の手がかりとなる要素を，非医療従事者が自然言語処理および機械学習手法を用いて発見し，闘病者や糖尿病予備軍の人々に助言できるシステムの構築を目指す． (Keyword) disease fighting blogs / lifestyle analysis / lifestyle-related disease / type 2 diabetes / text mining
36.	Norihide Kitaoka, Ryota Nishimura, 太田健吾, 石川晃之, 石川友香, 中野涼太, 田中五大, 鍋倉彩那, 佐藤辰耶, 渡邊凌太 and 大須賀晋 : Response control of 3D CG Agent Saya in spoken dialog with human, 日本音響学会研究発表会講演論文集, Vol.3-3-14, 1277-1280, Sep. 2021.
37.	Ryota Nishimura, Takahiro Mori and Norihide Kitaoka : Developing a Spoken Dialog System with Real-time Control Using ROS, 日本音響学会研究発表会講演論文集, Vol.2-3Q-4, 1123-1126, Sep. 2021. (Summary) 近年，音声対話システムがパソコンやスマートフォンなどに標準搭載されるようになり，その存在は大変身近なものとなってきている．しかし，これらの音声対話システムにおいては，実際の人間同士の会話で見られる相槌やオーバーラップなどの応答は未だ実現されていない. ユーザ発話に対してシステムから割り込んで応答を行うような対話を実現させるためには，リアルタイムの応答生成・応答タイミング決定を行うことが可能な音声対話システムを構築する必要がある．つまり，現在普及しているシステムでは，システムの設計から根本的に見直す必要があり，あいづちやオーバーラップを伴う対話の実現は困難である．本研究で構築した音声対話システムは，構成モジュールが独立し，並列に動作する．さらにそれらを密にリアルタイムに連携制御するために，モジュール間の通信にはROS のアーキテクチャを導入した．また，対話内容の表示や音声解析結果のグラフィカルな表示などに対応したダッシュボードも構築した．
38.	Meiko Fukuda, Ryota Nishimura, Nishizaki Hiromitsu, Iribe Yurie, Yamamoto Kazumasa and Norihide Kitaoka : Acoustic characteristics of the super-elderlies in the super-elderly speech corpus EARS, 日本音響学会研究発表会講演論文集, Vol.2-3P-11, 1105-1108, Sep. 2021. (Summary) 近年，介護の人手不足を背景とした高齢者とコミュニケーションができる介護ロボットのニーズが高まっている．また音声認識は特に手指や視力が不自由な高齢者にとって，スマートフォンなどの操作に便利なモダリティであり，高齢者音声の認識精度の一層の向上が求められている．しかし，高齢者の音声認識精度は青年・中年期のものと比べてword error rate (WER) が低下することが報告されている [1, 2, 3]．これらの背景から，我々は高齢者音声の音響モデル作成に利用するために，特に 80 歳以上の話者を多く擁する高齢者音声の大規模コーパス(EARS:Elderly Adults Read Speech) を構築中である [4]．高齢者音声は青年・中年のそれと比べ多様な変化がみられるが [5, 6, 7]，日本語話者の超高齢者についての報告は多くない．そこで本稿では超高齢者コーパスである EARS ならびに高齢者音声コーパス S-JNAS 話者(60-98 歳)の音響的特徴として，話速および母音継続長を測定した．また，EARS を利用した高齢者音声認識(HMM-DNN および Transformer ベースの End-to-end)の精度向上の簡便な方法として，EARS 音声に加工を施さずにそのまま複数回繰り返して音響モデル作成に使用した結果を報告する．
39.	Mori Hiroki, 太田健吾, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : Language model replacement method for Encoder-Decoder speech recognition model which excludes im- plicit linguistic information, 日本音響学会研究発表会講演論文集, Vol.1-3-1, 843-846, Sep. 2021.
40.	Horii Koharu, Meiko Fukuda, Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : End-to-End Spontaneous Speech Recognition Considering Hesitation, 日本音響学会研究発表会講演論文集, Vol.1-3-3, 851-854, Sep. 2021.
41.	Byambadorj Zolzaya, Ryota Nishimura, Ayush Altangerel, Ohta Kengo and Norihide Kitaoka : Cross-lingual, multi-speaker text-to-speech synthesis for low resource languages, 日本音響学会研究発表会講演論文集, Vol.1-3-7, Sep. 2021.
42.	Narangerel Purevdorj, Ryota Nishimura, Ayush Altangerel, Ohta Kengo and Norihide Kitaoka : How language similarity affects the Mongolian ASR using cross-lingual transfer learning, 日本音響学会研究発表会講演論文集, Vol.2-3-7, Sep. 2021.
43.	森大輝, 太田健吾, Ryota Nishimura, 小川厚徳 and Norihide Kitaoka : Language Model replacement method for end-to-end speech recognition which excludes implicit linguistic information, IPSJ SIG Technical Report, Vol.2021-SLP-137, No.17, 1-6, Jun. 2021. (Summary) Recently, end-to-end speech recognition has attracted much attention because it is faster and more concise than conventional DNN-HMM speech recognition. It has also been reported that recognition performance is improved by employing a language model trained with a large amount of text data. Based on these observations, we propose a new language model integration method which we call Language Model Replacement. In our proposed method, we use a pre-trained speech recognition model and a pre-trained language model. In contrast to the Shallow Fusion method, our proposed method can replace the linguistic information implied in the ASR model with independently trained model based on Bayes' rule. In our experiments, the ASR linguistic information implicitly trained using the Japanese language Academic Presentation Speech corpus is replaced with the language model trained using the Japanese language Simulated Public Speech corpus. We then compare ASR performance for Japanese speech recognition tasks using the Character Error Rate (CER). Our proposed Language Model Replacement method achieved 1.3 percent lower CER in comparison to the Shallow Fusion method.
44.	堀井こはる, Meiko Fukuda, 太田健吾, Ryota Nishimura and Norihide Kitaoka : End-to-End Speech Recognition of Non-fluent Speech by Hesitation Labeling, IPSJ SIG Technical Report, Vol.2021-SLP-137, No.17, 1-7, Jun. 2021. (Summary) Conventional ASR systems show high accuracy for fluent speech, but the accuracy is low for non-fluent speech such as that of the elderly. In this study, we experimented with end-to-end speech recognition to see how the accuracy of non-fluent speech is affected by labeling the speech as "stilted" and making it a target of recognition. The results showed that the sentence error rate was improved in all the evaluated data, and the results were valid, indicating that the model can learn the meaning of the labels.
45.	PUREVDORJ NARANGEREL, Ryota Nishimura, Ayush Altangerel, 太田健吾 and Norihide Kitaoka : Building low-resource speech recognizer: Transfer learning and data augmentation, 日本音響学会研究発表会講演論文集, Vol.3-2-9, 895-898, Mar. 2021. (Summary) Sequence-to-sequence (S2S) models are now widely used for end-to-end speech processing, especially in Automatic Speech Recognition (ASR) applications. But recently, a hybrid attention/CTC architecture which uses selfattention to model temporal context information has achieved significantly lower Word Error Rates (WER) [4] than S2S based systems. While attention-based encoder-decoder architectures are used in the best performing end-to-end ASR systems, these approaches cannot be easily adapted for low-resource ASR [3], for use with languages that lack large, well-annotated speech corpora. Our main goal in this paper is the development of a low-resource Mongolian ASR system. The training dataset we used contains 23 hours of continuous Mongolian speech from 217 speakers, which is much less data than what is normally needed to train conventional ASR systems. Lowresource ASR approaches typically rely on data augmentation (DA), which can be implemented at low cost. In this study, we also evaluate several multilingual transfer learning methods that use foreign language corpora to supplement lowresource target language training. We tested multiple DA and multilingual training approaches and compared their effectiveness using ASR character and word error rates (CER/WER) to measure performance.
46.	BYAMBADORJ ZOLZAYA, Ryota Nishimura, Ayush Altangerel, 太田健吾 and Norihide Kitaoka : Text to speech system for low resource languages by cross-lingual transfer learning and data augmentation, 日本音響学会研究発表会講演論文集, Vol.3-2-10, 899-902, Mar. 2021. (Summary) In this paper we proposed various TTS systems which contain both a spectrogram prediction network and a neural vocoder, for use when only a small amount of target data is available. We trained some models using only transfer learning and some using only data augmentation, to test how each method affected the naturalness of the output of the TTS model. However, we found that training the TTS model using both methods improved performance, reducing the gap between our low-resource model and the baseline M-MN model, which was trained with a larger amount of original target speech data. We also trained the Parallel WaveGAN vocoder using the same augmented data. As a result, our proposed method achieved almost the same speech quality as the vocoder trained with the entire corpus of target language data.
47.	Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Developing a Japanese End-to-End Speech Synthesis Server Considering Accent Phrases, 日本音響学会研究発表会講演論文集, Vol.1-2-7, 797-798, Mar. 2021. (Summary) Natural speech synthesis is one of the most important functions for interactive robots and information guidance systems. In order to generate high-quality synthesized voice, it is desirable to be able to use a state-of-the-art end-to-end speech synthesizer that assumes abundant computational resources, but it is not realistic to drive such a speech synthesizer on devices with limited computational resources such as robots, IoT devices, and smartphones. To solve this problem, cloud-based speech synthesis APIs such as Google Cloud's Text-to-Speech API1 and Amazon Polly API2 can be used, but they are not suitable for running robots and systems in environments where web access is not available. However, it is not suitable for running robots and systems in environments without web access. Therefore, in this research, we propose a high-quality Japanese speech synthesis server that can run within a LAN. This server runs on a GPU machine, receives input text via socket communication, generates natural synthetic speech using an end-to-end speech synthesizer, and sends it back to the client. Since not only the waveform information of the synthesized speech but also the time information of each phoneme in the synthesized speech is transmitted, it can be used for lip-syncing of CG avatars [1] or humanoid robots. In addition to the phoneme sequence of the input text, this server also explicitly inputs the accent phrase information extracted by g2p to the Text2Mel speech synthesizer. This makes it possible to produce synthesized speech with more natural intonation than a standard end-to-end speech synthesizer that takes only phonetic strings as input. The speech synthesis server described in this paper will be released as open source software*3.
48.	Norihide Kitaoka, Ryota Nishimura, Ohta Kengo and Hara Sunao : Development of a multimodal dialog system with photorealistic CG agent, 日本音響学会研究発表会講演論文集, Vol.1-2-6, 793-796, Mar. 2021. (Summary) When we think of the future human-machine cooperative society, an important issue is how natural and easy it is for machines and humans to communicate with each other. We believe that having a machine-side interface that looks as close to a human as possible and interacts as close to a human as possible is one of the ways to solve these problems. Therefore, we are focusing on "Saya," a 3D computer graphics system that is indistinguishable from a real human being and operates realistically in video images, and are constructing a system that can perform voice and multimodal dialogue using Saya as an agent. In this paper, we introduce the whole picture of Saya and the dialogue system we have developed, and give an overview of the speech recognition, speech synthesis, image recognition, and interaction control technologies used in the system.
49.	Meiko Fukuda, Iribe Yurie, Nishizaki Hiromitsu, Yamamoto Kazumasa, Ryota Nishimura and Norihide Kitaoka : Construction of EARS, a Speech Corpus for the Elderly, and Preliminary Study of Its Application to Speech Recognition, IPSJ SIG Technical Report, Vol.2020-SLP-1, No.6, 1-6, Oct. 2020. (Summary) Since the speech of the elderly has several features different from those of the general population, the recognition accuracy of the elderly is currently inadequate. In order to improve the accuracy, a large amount of speech data of the elderly is necessary. The S-JNAS, which has an average speaker age of 67.6 years, is widely used as a large-scale corpus of elderly speech. The S-JNAS, which has an average age of 67.6 years, has been widely used as a large-scale corpus of speech for the elderly. However, since there is a large age difference between the average life expectancy of the elderly in Japan and that of the Japanese population, we have started to construct a corpus of speech for the very elderly (EARS: Elderly Adults Read Speech). The design of the corpus is based on S-JNAS, and to date, we have collected and compiled a database of the speech of 121 people (average age: 83.4 years). In this paper, we describe the specifications of the corpus, and also report a preliminary study of the acoustic model of elderly speech using this corpus.
50.	hara kazuki, Kohei Torii, Kazuyuki Matsumoto, Minoru Yoshida, Ryota Nishimura, Jiro Tsuruki, 小林健一郎 and Kenji Kita : Tooth detection and segmentation from panoramic radiographs using Mask R-CNN, 令和2年度電気・電子・情報関係学会四国支部連合大会, No.15-9, Sep. 2020. (Summary) In Japan, the number of diagnostic items in dental diagnosis is considerably larger than in other countries, and it is necessary to reduce the burden of diagnosis on dentists because it is time-consuming for dentists to write dental formulae from scratch. Since interpretation of pantomime images is essential for diagnosis, understanding of dental health status, and planning of treatment, processing of pantomime images is an important issue for automation. In this study, tooth detection and segmentation from pantomime images are performed automatically. In this paper, we describe the method. (Keyword) medical image / deep learning
51.	fukuda tsubasa, Ryota Nishimura, Minoru Yoshida, Kazuyuki Matsumoto and Kenji Kita : A Study on the Analysis of Comics Using Deep Learning, 令和2年度電気・電子・情報関係学会四国支部連合大会講演論文集, Vol.15-7, 1, Sep. 2020. (Summary) The way we enjoy manga, a Japanese subculture, has been changing in recent years. In recent years, the way we enjoy manga, a Japanese subculture, has been changing. Looking at the sales trends in the market, we can see that e-books, which are read on electronic devices instead of paper books, are becoming popular. However, while smartphones and other small-screen devices are highly portable and easy to use, they are too small to display comic books, which are usually in paperback or A5 format, page by page. However, it is too small to display comics, which are usually in paperback or A5 size, page by page. If it is possible to extract and rearrange frames, it may provide an optimal reading environment for readers. In this study, we investigate the detection of coma regions to solve this problem.
52.	yamazaki taiga, Ryota Nishimura and Norihide Kitaoka : Construction of End-to-End Japanese Speech Synthesis System with Emotional Expression, 日本音響学会研究発表会講演論文集, Vol.2-P2-12, 859-862, Sep. 2020. (Summary) In this research, we develop a Japanese speech synthesis system that can express emotions using a deep learning-based method.In this study, we construct a Japanese speech synthesizer that can express emotions using a method based on deep learning. Using Tacotron2 as a reference, we aim to build a speech synthesizer with expressive power close to that of human voice. The final training data is the voice of Ms. Tomonatsu Toto from the statistical corpus of voice actors (10 minutes each of normal, happy, and angry speech). However, since this dataset alone is not enough to train Tacotron2, we solved the problem by training a model on a large-scale corpus (LJ Speech) and adding additional training to this model. In addition, we express emotion by assigning an emotion label to each character in the input text. The model constructed by this method can express not only one emotion for an entire text, but also can change the emotion in the middle of the text.
53.	(名) zolzaya, Ryota Nishimura, Ayush Altangerel and Norihide Kitaoka : NORMALIZATION OF TRANSLITERATED WORDS USING SEQ2SEQ MODEL WITH SPELL CHECKER, 言語処理学会発表論文集, Vol.E5-3, 1133-1136, Mar. 2020. (Summary) There are two written systems in Mongolian language: -Classic Mongolian (Uyghur Mongolian) -Cyrillic Both of them are used in Mongolia. The Mongolian People's Republic, as it was called then, first started using a modified Russian Cyrillic alphabet in 1940 which is still used and the official written system. Mongolian Cyrillic has 35 characters. Even though the official written system is the Cyrillic script before as mentioned, recently many people use Latin alphabets to write text on social media like Facebook and Twitter. While writing transliterated text using the Roman script on social media, there is no rule. Therefore, one word can be written in different forms. The text processing of social media is one of the important subjects in NLP. Therefore, in the last years, there has been a lot of work that focuses on social media. But there is a lack of research in this area for the Mongolian language and this is the first study of text normalization for Mongolian. Text normalization is a pre-processing stage for speech and language processing applications. At first, text normalization was to convert words in non-standard forms such as numbers, dates, acronyms, and abbreviations to standard forms in the formal text. But later this content was expanded to convert informal text on social media into formal text. Both source and target texts are the same languages in the most research work of noisy text normalizations. In our case, it is a little bit different and our purpose is to convert noisy transliterated text on social media to the formal style. In other words, the scripts of source and target texts are different, Roman and Cyrillic scripts, respectively.
54.	Mori Raita, Ryota Nishimura and Norihide Kitaoka : A spoken dialogue system with anaphoric analysis for spontaneous utterances, 日本音響学会研究発表会講演論文集, Vol.3-P-13, 1023-1026, Mar. 2020. (Summary) In this study, we proposed a dialogue system that complements free speech with a dialogue history, in which the subject and object are omitted. We proposed a method of completing free speech by using centralization theory, dependency analysis and case frames. In addition, we introduced input sentence shaping, which converts free-speaking spoken sentences into written sentences using the Sequence- to-Sequence model. Although the conversion from spoken to written words had a correct answer rate of about 40%, some sentences were supplemented with missing case particles, which may be useful in finding the antecedent of zero pronouns. We compared the output of the dialogue system to see if the system returned appropriate responses, and showed that speech completion is effective in a dialogue system.
55.	Obashi Yuya, Ryota Nishimura and Norihide Kitaoka : Evaluation of Language Model for Spontaneous Speech Recognition Trained Using Written Language-to-Spoken Language Text Conversion, 日本音響学会研究発表会講演論文集, Vol.2-Q-14, 951-954, Mar. 2020. (Summary) We converted a written text corpus to a spoken text corpus using a sequence-to-sequence model and trained a spoken language model. Although the accuracy of the conversion from written to spoken text was not very high, the learned language models were useful for spoken-speech recognition, as they captured the statistical characteristics of spoken words well. We also investigated what kind of spoken language features this language model is effective for. The results show that the present study is very effective for the insertion of filler words and spoken word-specific endings, and is also effective to some extent for colloquial expressions such as particle omissions.
56.	Meiko Fukuda, Nishizaki Hiromitsu, Iribe Yurie, Ryota Nishimura and Norihide Kitaoka : Construction of elderly Japanese speech corpus for acoustic modeling and analysis of the influence of age and dialects on elderly speech recognition, 日本音響学会研究発表会講演論文集, Vol.2-Q-13, 947-950, Mar. 2020. (Summary) In this study, acoustic models for elderly people's speech recognition were created using the existing corpus of speech corpus (JNAS, S-JNAS and CSJ) and the elderly people's speech in this corpus. The recognition accuracy was slightly improved in spite of the small amount of speech data in this corpus. WER tended to decrease with age. It was shown that each regional dialect has an effect on the recognition accuracy when learning an acoustic model. In the future, we would like to consider further improvement of the acoustic model by expanding the corpus, the impact of the regional dialects on the speech recognition, and an effective additional learning method for the acoustic models that were not obtained in this study.
57.	Jiahao CHEN, Ryota Nishimura and Norihide Kitaoka : Streaming speech recognition with Uni-directional LSTM and local attention, 日本音響学会研究発表会講演論文集, Vol.2-Q-12, 943-946, Mar. 2020. (Summary) In this paper, an end-to-end speech recognition model for streaming speech with unidirectional LSTM and local attention is proposed. We have confirmed that the introduction of local attention improves the recognition accuracy. We also confirmed that local attention is more effective for CTC output at high sub-sampling rates, which is important for streaming models. We believe that this is an important property for streaming models. In the future, we plan to improve the accuracy of out-of-domain recognition by extending the language model by taking advantage of the property of the attention mechanism.
58.	Goto Tomoaki, Ryota Nishimura and Kenji Kita : Mapping of acoustic vector sequence and document vector sequence, 情報処理学会全国大会講演論文集, No.4Q-05, 2-181-2-182, Mar. 2020. (Summary) In this study, we have constructed a model that allows for the mapping of acoustic features to document vectors. The proposed method uses RNN-LSTM for training time series data to enable cross-media mapping. We conducted evaluation experiments using the model and confirmed that the model mapped well in closed tests. In the future, we will conduct an open test to investigate the generalization performance.
59.	Kiyohara Kanta, Ryota Nishimura and Norihide Kitaoka : Construction of Geometric Problem Solving Assist Sys- tem Based on Integrative Understanding of Speech and Finger Pointing, 情報処理学会全国大会講演論文集, No.5F-03, 4-5-4-6, Mar. 2020. (Summary) In this paper, two modalities, speech and pointing, are treated as inputs, and the system is designed to solve geometric problems by integrating each modality. The evaluation experiment showed that the GUI was rated higher than the previous study, but the usability of the voice recognition system was rated lower. However, this system is suitable for investigating the relationship between speech and pointing because it does not limit the input grammar and allows users to give free answers. The future work is to implement a mechanism to compensate for misrecognition and to improve the ability to understand the input, so that the input can be more natural. We also plan to conduct an experiment to investigate the relationship between speech and pointing.
60.	Komatsu Akihisa, Ryota Nishimura and Norihide Kitaoka : Environmental Sound Recognition Using Convolutional-LSTM, 日本音響学会研究発表会講演論文集, Vol.2-Q-17, 925-928, Sep. 2019. (Summary) In this paper, we attempted the task of detecting multiple acoustic events in input speech using the machine learning models, CNN, CLSTM and LSTM models. The results showed a recognition accuracy of about 56 in F-values using the audioSet dataset. Furthermore, comparisons between the LSTM model and the CLSTM model were conducted, and the CLSTM model showed better and more consistent accuracy.
61.	Yamamoto Taiki, Ryota Nishimura and Misaki Masayuki : Small-footprint Magic Word Detection Using Convolutional LSTM, 日本音響学会研究発表会講演論文集, Vol.2-3-4, 819-822, Sep. 2019. (Summary) In this study, we propose a memory-saving MagicWord detection method using CLSTM. Compared with the model with LSTM only, the structure of CLSTM combined with LSTM or NIN achieves especially high performance. In particular, the clstm lstm nin 2 improves the baseline performance by 76.4%. A comparison between the structures before and after data expansion [8] shows that some of the CLSTM structures achieve higher performance with less data. In the future, we plan to improve the implementation of CLSTM and the structure of clstm lstm, and also to study a model that allows free configuration of MagicWord.
62.	Meiko Fukuda, Ryota Nishimura, Nishizaki Hiromitsu, Iribe Yurie and Norihide Kitaoka : A New Corpus of Elderly Japanese Speech for Acoustic Modeling, and a Preliminary Investigation of Dialect Dependent Speech Recognition, 日本音響学会研究発表会講演論文集, Vol.1-P-17, 875-878, Sep. 2019. (Summary) We planned to construct a corpus of audio data for an even older age group than S-JNAS, and have compiled a database of 221 audio data from 4 locations in Japan, with an average age of 79.2 years, based on Kaldi's CSJ recipe. The CSJ acoustic model was found to have the best recognition accuracy among the three methods. The CSJ acoustic model showed the best recognition accuracy.
63.	Jiahao CHEN, Ryota Nishimura and Norihide Kitaoka : E2E streaming speech recognition with CTC and Attention, 日本音響学会研究発表会講演論文集, Vol.1-P-16, 871-874, Sep. 2019. (Summary) In this study, an extension of the CTC model based on local attention is proposed for Japanese streaming speech recognition. For downsampling, maxpooling achieves better performance for CTC word recognition compared to conventional frame stacking. We also found that using local attention improves the performance at low frame rates, which is useful for low-resource streaming speech recognition. Finally, we are able to further reduce the WER with the confidence penalty. In the future, we would like to extend RNN-LM to eliminate the effect of blank and repetitive symbols.
64.	Obashi Yuya, Ryota Nishimura and Norihide Kitaoka : Text Conversion of Written Language into Spoken Language for Language Model Training for Spoken Language Automatic Speech Recognition System, 日本音響学会研究発表会講演論文集, Vol.1-3-8, 807-810, Sep. 2019. (Summary) In this study, we proposed to convert written text to spoken text using the Sequence-to-Sequence model. If we can do this accurately and efficiently, we can create a large corpus of spoken text to improve the accuracy of spoken speech recognition. We first created the corresponding written sentences manually using the corpus of elderly dialogue transcription and the corpus of nominal speech to create a bilingual dataset of written and spoken words. We constructed a transformation model and converted a part of the BCCWJ corpus into spoken language. Although the accuracy of the conversion from written to spoken language was not good, we were able to convert some sentences very well. It was also shown that the language model trained on the converted sentences gave better perplexity to the spoken language than the one trained on the untransformed sentences, even if the written sentences were not correctly converted to spoken ones. Therefore, we believe that language models learned with a spoken text corpus created by converting written text to spoken text are effective for spoken speech recognition. We plan to add examples of successful transformations of test data to the training data in order to increase the training data, and to create a more accurate and large-scale text corpus of spoken language by using the Seq2Seq model with an Attention mechanism to accurately transform long sentences.
65.	Kanta Kiyohara, Ryota Nishimura and Norihide Kitaoka : Development and Evaluation of Learning Support System for Geometric Problem Solving Using Finger Pointing and Speech, 日本音響学会研究発表会講演論文集, Vol.2-P-17, 989-992, Mar. 2019. (Summary) In this paper, we have constructed a system that can solve geometric problems by multi-modal interaction using voice and pointing. As a result, it is possible to draw a new auxiliary line in the problem figure, draw equal angles, and express them as mathematical formulas, and show the effectiveness of the proposed system through evaluation experiments. (Keyword) マルチモーダルインタラクション / 幾何問題
66.	Kazuaki Kajinami, Ryota Nishimura, Iribe Yurie and Norihide Kitaoka : Spoken dialog recording system for development of spoken dialog breakdown detection methods, 日本音響学会研究発表会講演論文集, Vol.2-P-2, 949-952, Mar. 2019. (Summary) In order to construct a voice dialogue database for the development of a voice dialogue failure detection method, voice dialogue experiments were conducted and voice data was recorded.The Wizard of Oz method was also used to prevent excessive dialogue failure. A voice dialogue system has been constructed and voice recording has been carried out. (Keyword) 対話破綻検出 / 音声対話
67.	Umehara Yasuyuki, Ryota Nishimura and Norihide Kitaoka : Construction of spoken dialogue system integrating various dialog systems, 日本音響学会研究発表会講演論文集, Vol.2-P-1, 945-948, Mar. 2019. (Summary) We proposed a dialogue system that included multiple dialogue systems. In the proposed dialogue system, we introduced a response sentence selection method that selects more natural response sentences from the generated response sentence candidates. Therefore, we proposed a method for scoring candidate sentences using a model trained with word vector representation as input and output. At present, we have not been able to obtain highly accurate results. (Keyword) Spoken Dialog System / 応答選択 / 対話戦略
68.	Ohta Kengo, Ryota Nishimura and Norihide Kitaoka : Selecting Type of Response for Chat-like Spoken Dialogue Systems using Multi-task Learning of LSTM, 日本音響学会研究発表会講演論文集, Vol.2-P-3, 953-956, Mar. 2019. (Summary) In order to improve the conversation continuation rate of the chat dialogue system, we proposed a method for selecting the response type of the system by a framework using the Encoder-Decoder model by LSTM. We conducted an evaluation experiment of the proposed method using one-to-one speech data of the elderly and interviewers. By performing multitask learning using a predictor of the response utterance and a predictor of the response type, High accuracy has been achieved compared to simply predicting only the response type. (Keyword) 応答種別選択 / 音声対話システム / LSTM / マルチタスク学習
69.	Kanta Kiyohara, Ryota Nishimura and Norihide Kitaoka : 指差しと口述説明を理解する幾何学問題学習支援システム, Forum on Information Technology 2018 (FIT2018), Vol.J-011, 1-2, Sep. 2018.
70.	Motoki SHIMOKASA, Hiromitsu Nishizaki, Meiko Fukuda, Ryota Nishimura and Norihide Kitaoka : A Study on Speech Recognition Models for Super Elderly Spontaneous Speech, 日本音響学会講演論文集, Vol.1-R-10, 977-978, Sep. 2018. (Summary) 本稿では，超高齢者の自然発話音声のための音声認識モデルの検討を行う．認識実験の結果，評価セットの音声とドメインが一致するコーパスから音響モデルを学習するのがよいことが明らかとなった．
71.	Yuya Obashi, Ryota Nishimura and Norihide Kitaoka : Mining OOV words from vocabulary shift in Twitter and linguistic adaptation of speech recognizer to the OOV words, 日本音響学会講演論文集, Vol.1-R-24, 1017-1020, Sep. 2018. (Summary) 本稿では，Twitterを用いて新しく使用されるようになった音声認識システムにとって正しく認識されないと考えられる未知語の発見手法の提案と，発見した未知語に対応した言語モデルを作成し，音声認識システムに適用する手法の提案を行う．Twitter から取得したツイートを，新しいツイートと過去のツイートとで比較することにより，ユーザー間の使用単語の変化が取得でき，そこから新しく使用されるようになった未知語の取得が可能となる．提案手法を用いて作成した言語モデルでは，ベースラインと比較して補正パープレキシティの改善が得られ，取得した未知語が含まれる評価セットにおいてパープレキシティ及び補正パープレキシティの改善が得られた．音声認識実験では，作成した言語モデルを適用した音声認識システムで未知語が含まれる文を認識した場合，音声認識システム全てにおいて，ベースライン音声認識システムより良い単語誤り率と未知語誤り率を得られた．ベースラインシステムと比較して相対的に約 91%の未知語誤り率の改善が得られた．これによって，未知語誤り率の改善が全体的な単語誤り率の改善へと繋がる結果となった．
72.	Taiki Yamamoto, Ryota Nishimura, Masayuki Misaki and Norihide Kitaoka : Magic word detection method in continuous speech using LSTM neural network, 日本音響学会講演論文集, Vol.1-R-21, 1009-1012, Sep. 2018. (Summary) 本論文では, LSTM を用いた Magic Word 検出手法を提案する. BPTT 法により, Convolutional LSTM 及び, Network in network を用いた提案モデルの学習・評価を行う. ベースライン手法と比較して, 提案モデル conv16 clstm16 は非常に良い性能を発揮した. 結果より, CLSTM 構造が Magic Word 検出において有意であることが示された. また, 時空間を結びつけた特徴を捉える事が Magic Word 検出において有効である事が得られた.
73.	Ryota Nishimura, Miho Higaki and Norihide Kitaoka : RNNに基づく音響ベクトル時系列の文書ベクトルへのマッピング, IEICE Technical Report, No.SP2018-2, 1-6, Jun. 2018. (Summary) 音楽の音響データと歌詞のデータをニューラルネットワークによって対応付ける．音響データをクロマベクトル系列にしてLSTMでベクトルに変換する．その変換先が歌詞ベクトル(Word2vecによる歌詞中の単語のベクトル化したものの平均)になるように学習し，対応付け可能であることを示した．
74.	Kazuaki Kajinami, Ryota Nishimura and Norihide Kitaoka : Construction of dialogue database for development of spoken dialogue breakdown detection methods, 日本音響学会講演論文集, Vol.2-Q-14, 177-180, Mar. 2018. (Summary) 音声対話破綻検出手法の開発に向けた音声対話データベースを構築するために音声対話実験を行い，音声データの収集を行った．ユーザと音声対話システムが音声対話を行うことで対話文脈から対話破綻が引き起こされる場合，音声認識による誤認識のために対話破綻を引き起こす場合などさまざまな要因によって対話破綻が起こるという結果となった．音声の誤認識による対話破綻は言語情報に基づく対話破綻検出では検出することは難しく，音声対話であることに起因する破綻の要因を知ることができた．
75.	Ryota Nishimura, 陳伯翰 and Norihide Kitaoka : OOV registration to language model for speech recognition, 日本音響学会講演論文集, Vol.1-Q-23, 127-130, Mar. 2018. (Summary) 音声認識では事前に認識辞書に登録した単語は認識できるがそれ以外の未知語は一般に認識できない．しかし音声認識のタスクでは認識辞書にない単語を認識しなければならない場合が多数ある．そこで，言語モデルの学習時に未知語を登録するための「未知語トークン」を織り交ぜたデータベースを用い，認識時に未知語トークンを認識したい単語に置き換える簡単で有効な未知語登録手法を提案した．
76.	太田健吾, Ryota Nishimura and Norihide Kitaoka : Selecting Type of Response for Chat-like Spoken Dialogue Systems Based on LSTM-RNN Considering Order of Words, 日本音響学会講演論文集, Vol.2-8-7, 45-48, Mar. 2018. (Summary) 雑談音声対話システムを構築することを考えた場合，相手の発話などから，次にどのような発話をすればよいか発話種別選択が必要になる．相手の発話の音響的特徴に基づいて，LSTMを用いて，相槌や発話健康体などの発話種別の選択を可能とする方法を提案した．
77.	Takuma Nakagawa, Ryota Nishimura, 入部百合絵, 石黒祥生, 大須賀晋 and Norihide Kitaoka : 自動運転車の操作におけるマルチモーダルインタラクション, 日本音響学会講演論文集, 1-4, Mar. 2018. (Summary) 将来の自動運転車において，タクシー運転手に指示するかのように操作できるインタフェースを考えた．音声対話に指差しや視線の情報を統合して，交差点を曲がったり停車する場所を指定するなど様々な操作を自然な方法で実現できるようにした．
78.	Seto Eichi, Ryota Nishimura and Norihide Kitaoka : 単語の分散表現に基づく事例ベース雑談音声対話システムのユーザ適応, 日本音響学会講演論文集, Vol.2-8-8, 49-52, Mar. 2018. (Summary) 事例ベースの音声対話システムにおいて，限られた事例を，ユーザ固有の適切な事例に自動変換することにより，音声対話システムをユーザに適応する．ユーザのプロフィール情報からWeb検索によって得られた関連語を事例の単語と置き換える方法でユーザ適応を可能にした．
79.	太田健吾, Ryota Nishimura and Norihide Kitaoka : Selecting Response from Conversational Spoken Dialogue System Based on Distributed Representation of User Utterances, IEICE Technical Report, No.SP2017-55, 1-5, Dec. 2017. (Summary) 雑談音声対話システムを構築することを考えた場合，相手の発話などから，次にどのような発話をすればよいか発話種別選択が必要になる．相手の発話の言語情報を，単語の分散表現で表現することで，相槌や発話健康体などの発話種別の選択を可能とする方法を提案した．
80.	Eichi Seto, Ryota Nishimura and Norihide Kitaoka : User adaptation of examples for example-based reminiscence therapy spoken dialog system using word embedding, IEICE Technical Report, Vol.117, 23-28, Oct. 2017. (Summary) 認知症の予防・改善のために自然な雑談対話(回想法)を行う音声対話システムを開発する必要がある．事例ベースの対話システムにおいて，ユーザのプロフィール情報を用いた対話事例のユーザ適応法を提案する．情報検索により取得したユーザと関連が深い単語と対話事例の単語間の cos 類似度を word2vec によって計算し，閾値以上の単語を置換することで，ユーザの話題に沿った事例を事前に生成する．本研究では，生成された事例の自然さを人手で評価することで，手法の有効性を示した．また，word2vec による分散表現の加減算を用いることで，類似度のみの場合に生成されていた不自然な事例の一部を自然な事例にすることができた．
81.	Takahiro Hirano, Takahiro Uchiya, Ichi Takumi, Ryota Nishimura and Masaru Sakurai : Proposal ofreminiscence system to suppress dementia by voice dialogue, 情報処理学会全国大会講演論文集, No.4ZB-03, 571-572, Mar. 2017. (Summary) 近年，認知症患者は増加し続けており，介護者の負担は増大しつづけている．しかし，認知症の治療法は未だ確立されておらず，いかに進行を抑制するかということが認知症治療の課題となっている．認知症の進行を抑制する手段の一つに回想法がある．回想法とは患者と介護者が対話を行う治療法である．しかし，回想法は認知症患者と何度も対話を行う必要があり，介護者の負担が大きいという問題点がある．そこで本研究では音声対話システムであるMMDAgent[1] を用い，介護者の代わりに回想法を行うシステムを提案する．
82.	Ryota Nishimura, Rintaro Manabe and Yukiko Nakano : Development of Spoken Dialogue System which Integrates and Shares Information based on ROS Architecture, 人工知能学会研究会資料, Vol.B5, No.03, 79-84, Mar. 2017. (Summary) 本稿では，ROSアーキテクチャにて動作する，音声対話システムについて述べた．音声対話システムに必要な 5 つのモジュール(音声認識，言語理解，対話管理，応答生成，音声合成)を構築し，さらに情報統合を目的とした Information State モジュールを導入した． ROSアーキテクチャを導入したことにより，通信路確保，データ形式の統一を円滑に図ることができたため，提案システムのような複数モジュールを活用した音声対話システムが実現できた．現在は，喫茶店実践，教育実践などの実践アプリケーションを構築中であり，複数モジュールが連携して動作する中で，音声対話システムも活用され，本稿で提案した Information State がデータ共有の中心モジュールとして活用されている．
83.	Takeshi Morita, Ryota Nishimura and Takahira Yamaguchi : PRINTEPS: A Total Intelligent Application Development Platform based on ROS and Its Application, 人工知能学会知識ベースシステム研究会, Vol.108, 30-35, Jun. 2016. (Summary) 本稿では，ROSに基づく総合知能アプリケーション開発プラットフォームPRINTEPSのアーキテクチャ， PRINTEPSにおけるワークフローエディタ，ケーススタディとしてロボット喫茶店における入店時挨拶について述べる．ワークフローエディタは，「http://printeps.org」にて，公開している．
84.	Takeshi Morita, Ryota Nishimura and Takahira Yamaguchi : The Architecture of PRINTEPS: A Total Intelligent Application Development Platform based on ROS, 人工知能学会全国大会, Vol.4C4-6, 1-4, Jun. 2016. (Summary) 本稿では，ROS に基づく総合知能アプリケーション開発プラットフォーム PRINTEPS のアーキテクチャ，PRINTEPS におけるワークフローエディタ，PRINTEPS の評価指標について述べる．ワークフローエディタは，「http://printeps.org」にて，公開している．
85.	Ryota Nishimura, Yutaka Takase and Yukiko Nakano : Development Environment of Spoken Dialogue System Based on PRINTEPS, 人工知能学会全国大会, Vol.4C4-2, 1-4, Jun. 2016. (Summary) 本稿では，PRINTEPSアーキテクチャにて動作する，音声対話システムの開発について述べた．音声対話システムに必要な 5 つのモジュール(音声認識，言語理解，対話管理，応答生成，音声合成)を構築し，これらを組み合わせて，音声対話システムを構成している．PRINTEPSからは，対話から得たい情報(対話したい内容:対話ゴール)を指定して音声対話システムを呼び出すことで，音声対話システムはユーザと音声対話をし，必要な情報を取得することができる．各対話ゴールに対応する細かい対話処理ルールなどは，予め用意し，PRINTEPSによる実践知能アプリケーション開発者は，音声対話に関する知識なしに，音声対話システムを構築・利用することが可能である．
86.	Rintaro Manabe, Yutaka Takase, Ryota Nishimura and Yukiko Nakano : User modeling for Robot Café Interaction in PRINTEPS, 人工知能学会全国大会論文集, Vol.4C4-4, 1-4, Jun. 2016. (Summary) 本稿では，喫茶店という状況に焦点を当て，ユーザモデルに基づきコミュニケーションを変更する社会的ロボットを実装するために，ロボットが認識すべきユーザ情報について考察する．また，接客対話の収集と分析を行い，ロボットが社会的シグナルである視線や着席行動を認識できれば，社会的関係や，親密度を推定することが可能であることが示された．そして，現在実装中のプロトタイプシステムについて記述し，システムの問題点について述べる．
87.	Takeshi Morita, Ryota Nishimura and Takahira Yamaguchi : PRINTEPS: Total Intelligent Application Development Tool based on Stream Reasoning and ROS, IEICE Technical Report, Vol.115, No.375, 55-60, Dec. 2015. (Summary) 本稿では，ストリーム推論と ROS に基づく総合知能アプリケーション開発ツール PRINTEPS(PRactical INTElligent aPplicationS)を提案する．本研究では，PRINTEPS のケーススタディとして，Pepper を接客用ロボットとして用いた喫茶店業務実践を取り上げる．特に，ストリーム推論と ROS を用いた来客検知および喫茶店サービスオントロジーとルールに基づく入店時挨拶について紹介する．
88.	Ryuhei Kani, Takahiro Uchiya, Daisuke Yamamoto, Ichi Takumi and Ryota Nishimura : Development of the Communication Mechanism among Voice-Interactive Digital Signages, 平成27年度電気・電子・情報関係学会東海支部連合大会講演論文集, Vol.D3-5, 1, Sep. 2015. (Summary) 音声対話デジタルサイネージ間の通信機構を提案した．これにより，今までは情報を受け取るだけであった利用者が情報を発信したり，メイちゃんの継続的利用などによる利用者支援が可能になる．
89.	Takahiro Kobayashi, Ichi Takumi, Arao Funase, Takahiro Uchiya and Ryota Nishimura : Improvement of successful speech recognition on MMDAgent used as Virtual Telephone Operator, 平成27年度電気・電子・情報関係学会東海支部連合大会講演論文集, Vol.D3-4, 1, Sep. 2015. (Summary) MMDAgentを用いた電話交換手システムにおける音声の誤認識に対する改善案として，DTMF 音の認識機能の追加に加えて文脈に応じた辞書ファイル切り替えを提案した．
90.	Ryuhei Kani, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Prototype of the Communication Mechanism among Voice-Interactive Digital Signages, マルチメディア,分散,協調とモバイル(DICOMO2015)シンポジウム講演論文集, Vol.6F-1, 1295-1300, Jul. 2015. (Summary) 本学では，デジタルサイネージでの情報取得において，3D キャラクタとのリアルタイムでの音声対話が可能な音声ツールキットとして MMDAgent が利用されている．MMDAgent は独立したシステムとして設計されているため，従来では複数端末間における連携を行うことが不可能である．本研究では，この問題点を解決するため，複数サイネージ間での協調動作を支援する通信機構を提案する．
91.	仲野良佑, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : エージェントを用いたスマートフォン向け音声対話システム拡張機構の開発と評価, IPSJ SIG Technical Report, Vol.2015-ICS-179, No.2, 1-8, Mar. 2015. (Summary) 3D キャラクタとのリアルタイムでの音声対話が可能な音声対話システムとしてMMDAgent が提案されている．この動作環境を拡大し，利便性を向上させるためにMMDAgent を Android スマートフォン上へ移植したスマートメイちゃんが提案されている．これらのツールキットにおいて採用されているシナリオ定義手法では，複雑な対話シナリオの作成や，端末間通信による連携を必要とするシナリオの構築が困難である．本研究では，これらの問題点を解決するため，スマートメイちゃんをソフトウェアエージェントにより拡張する機構を提案，実装し，有効性を確認するための評価実験を行った．エージェントを用いてスマートメイちゃんの拡張を行うことで，複雑な応答文や条件分岐を持つシナリオの開発がより容易となり，また，ネットワークと連携する必要があるシナリオの開発も，エージェント間通信の延長線上と捉えることで格段に容易となる．評価実験の結果，提案機構によって問題点を解決できたことを確認したが，実用的な機構とするためにはいくつかの課題点も存在することが確認された．
92.	大竹聡史, 打矢隆弘, 山本大介, Ryota Nishimura and 内匠逸 : Extraction of User Preference based on Voice Interaction, 平成26年度電気・電子・情報関係学会東海支部連合大会講演論文集, Vol.2014, L1-6, Sep. 2014. (Summary) 音声対話システムを用いた嗜好抽出技術について提案した．
93.	Yuichi Matsushita, Takahiro Uchiya, Ryota Nishimura, Daisuke Yamamoto and Ichi Takumi : Construction of Creation Environment of Voice Interaction Scenario, 平成26年度電気・電子・情報関係学会東海支部連合大会講演論文集, Vol.2014, K1-5, Sep. 2014. (Summary) 大人数での対話シナリオの作成を行うための機構として，クラウド型対話シナリオ作成環境の構築を提案した．
94.	藤堂祐樹, Ryota Nishimura, 山本一公 and 中川聖一 : Impressive Comparison of The Human Voice and Synthetic Speech in Response Sentence of Spoken Dialogue System, 日本音響学会秋季研究発表会, Vol.2-8-9, 79-83, Sep. 2013. (Summary) 本報告では，対話システムにおける応答文の合成音声と人間の録音音声について，ユーザに与える印象・満足度について調査を行った．被験者実験の結果，録音音声はエージェントのグラフィックの質と多少の差異があったとしても，エージェントへの親しみや，話しやすさの印象を被験者に与えることが示された．
95.	打矢隆弘, 山本大介, Ryota Nishimura, 内匠逸, 大浦圭一郎, 李晃伸 and 徳田恵一 : Development and Experiment of Voice Interaction Agent based on Video/Audio Streaming Technology, 平成25年度電気関係学会東海支部連合大会講演論文集, Vol.2013, S1-3, Sep. 2013. (Summary) 本研究では，映像音声ストリーミング技術を用いて，利用場所を問わずに 3D キャラクタと音声対話を行うことが可能な，ビデオ通話型音声対話エージェント「モバイルメイちゃん」を提案した．また各種の実証実験により，提案システムの有効性を確認した．
96.	山本大介, 大浦圭一郎, Ryota Nishimura, 打矢隆弘, 内匠逸, 李晃伸 and 徳田恵一 : An Effort for Stand-alone Mobile Voice Interaction Agents on Smartphones, 平成25年度電気関係学会東海支部連合大会講演論文集, Vol.2013, S1-4, Sep. 2013. (Summary) 本研究では，スマートフォン単体で動作可能な音声対話3D エージェントシステムを開発した．従来のスマートフォン向け音声対話システムとは違って，遅延の少ないより自然な音声対話が可能になった．今後の課題は，よりネットワークと連携した音声対話システムの構築の仕組みを検討していく
97.	Ryota Nishimura, 山本大介, 打矢隆弘 and 内匠逸 : Development of Dialog Scenario Editor aimed at popularization of Spoken Dialog System, マルチメディア, 分散協調とモバイルシンポジウム 2013 論文集, Vol.2013, 1796-1799, Jul. 2013. (Summary) 本論文では，音声対話エージェントのシナリオ構築環境を改善するべく，シナリオエディタの開発を行う．このシナリオエディタを用いることで，対話シナリオを見やすくし，編集しやすくして，より簡単に扱うことができるようにするのが目的である．開発したシナリオエディタには，大きく 3 つの特徴がある．1 つ目の特徴は，様々なプラットフォームでの動作が可能ということである．MMDAgent 自体も， PC(windows，mac，Linux)や，スマートフォン(android)など様々なプラットフォームに移植されており，シナリオエディタについても，同様の環境での需要が考えられる．これらの環境で広く動作させる為，web ブラウザベースでの開発を行なっている．
98.	山本大介, Ryota Nishimura, 打矢隆弘 and 内匠逸 : Smart Mei-chan: A Mobile Voice Interaction Agent using Smartphone Functions, マルチメディア, 分散協調とモバイルシンポジウム 2013 論文集, Vol.2013, No.2B-6, 300-304, Jul. 2013. (Summary) 本稿では，音声対話システム構築ツールキット MMDAgent を拡張し，GPS や NFC・GUI 等の Android スマートフォンの機能を効果的に活用するための仕組みを提案する．具体的には，全ての Android OSの機能を利用するためには Java 言語で開発する必要があるが，MMDAgent は C++言語での開発が想定されており，異なる言語間での連携は容易ではないという問題があった．そこで，Java と C++の連携を実現するブリッジモジュールを開発し，容易にMMDAgent の各モジュールと Android の機能とを連携可能な Java プラグイン開発プラットフォームを提案した．また，本プラットフォームに基づく様々な Java プラグインの開発も行った．
99.	山本大介, 大浦圭一郎, Ryota Nishimura, 打矢隆弘, 内匠逸, 李晃伸 and 徳田恵一 : Development of a Voice Interaction Agent using 3D Modeling for Stand-alone Smartphones, 情報処理学会インタラクション, No.3EXB-27, 675-680, Mar. 2013. (Summary) 本研究では，スマートフォン単体で動作する音声対話3Dエージェントシステムを開発した．従来のスマートフォン向け音声対話システムとは違って，提案システムでは，スマートフォン内で音声認識や音声合成などの処理を完結することができるため，ネットワークに起因する遅延のない自然な音声対話が実現可能になった．さらに，スマートフォンに適した音声インタフェースを試作し，提案手法の評価を行った．
100.	藤堂祐樹, Ryota Nishimura, 山本一公 and 中川聖一 : Analysis and Evaluation of Spoken Dialog System with Multiple Agents, IPSJ SIG Technical Report, Vol.2013-SLP-9, No.18, 1-7, Feb. 2013. (Summary) 現在のほとんどの音声対話システムは，ユーザとシステムの1対1の対話を扱っているが，我々はシステム側のエージェントを2人にした三者対話システムの開発を行ってきた．対話ドメインは従来の「うどんとラーメンのどちらが好きか」の他に「北海道旅行と沖縄旅行のどちらに行きたいか」を追加し，それぞれ同じドメインの二者対話システムと三者対話システムをユーザに使用し，評価実験を行った．被験者実験の結果，三者対話システムは，対話の弾み具合の向上や，対話の雑談らしさの印象を被験者に与えるという結果が指示された．
101.	打矢隆弘, 山本大介, 柴川元宏, 吉田真基, Ryota Nishimura, 内匠逸 and 松尾啓志 : オープンキャンパスにおけるモバイル端末向け音声対話システムの活用, 大学ICT推進協議会年次大会2012, No.H3-4, 1-8, Dec. 2012. (Summary) 本研究ではオープンキャンパスにおいて高校生に学科案内やキャンパス案内を行う音声対話システムを開発した．このシステムはスマートフォンとVoIPを用いて利用場所を問わずに音声対話が利用可能であり，オープンキャンパス参加中の建物案内や大学周辺の地理案内に最適である．また，利用者は大学に関する情報を音声で気軽に入手することができる．本稿では，今年度のオープンキャンパスでの利用実績とユーザビリティの評価実験結果について報告する．
102.	打矢隆弘, 山本大介, 柴川元宏, 吉田真基, Ryota Nishimura and 内匠逸 : Development of spoken dialogue service based on video call named Mobile Meichan, 合同エージェントワークショップ&シンポジウム2012(JAWS2012)予稿集, 1-7, Oct. 2012. (Summary) 本研究では，スマートフォンと VoIP を用いて利用場所を問わずに音声対話を行うことが可能な，ビデオ通話型音声対話サービス「モバイルメイちゃん」を提案した．また各種の実験により，提案サービスの有効性を実証した．
103.	藤堂祐樹, Ryota Nishimura, 山本一公 and 中川聖一 : Analysis and Evaluation of Spoken Dialog System with One Agent and Multiple Agents, IPSJ SIG Technical Report, Vol.2012-SLP-90, No.19, 1-8, Feb. 2012. (Summary) 現在のほとんどの音声対話システムは，ユーザとシステムの 1 対 1 の対話を扱っているが，本報告ではシステム側のエージェントを 2 つにした三者対話システムの開発を行った．また二者対話システムと三者対話システムをそれぞれユーザに使用してもらい，システムがユーザに与える影響・満足度についての分析を行った．対話ドメインは「うどんとラーメンのどちらが好きか」とし，二者対話システムのエージェントにはユーザにうどんとラーメン両方を薦めさせた．三者対話システムのエージェントには，それぞれうどん好き，ラーメン好きという個性を与え，それぞれ自分の好きな物をユーザに薦める対話形式とした．被験者実験の結果，三者対話システムは，エージェントへの親しみや対話の雑談らしさの印象を被験者に与えることが示された．
104.	Ryota Nishimura and 駒谷和範 : Estimation of Dialogue States in Database Search Spoken Dialogue System, IPSJ SIG Technical Report, Vol.2012-SLP-90, No.20, 1-7, Feb. 2012. (Summary) データベース検索を行う音声対話システムにおいて，ユーザの意図を反映し，音声認識誤りに対処して応答生成を行うための対話状態を推定するモデルについて述べる．データベース検索タスクにおいて，対話の状態が「検索条件の指定」「情報の提示要求」の二つを遷移するとモデル化する．この 2 つの状態を対話中から得られる素性に基づき，ロジスティック回帰により予測する．レストランデータベース検索を行う音声対話システムを構築し，7 名の被験者から対話データを収集し，モデルの学習実験を行った．ベースラインシステムでの対話状態の決定精度が 87.1%であるのに対して，学習されたモデルでは，オープンテストで 97.4%であった．また，モデルに用いた素性のうち，どの素性が対話状態の推定に寄与しているかの確認も行った．
105.	Ryota Nishimura and 中川聖一 : Development and Evaluation of Spoken Dialog System using Multiple Dialog Agents, IPSJ SIG Technical Report, Vol.2010-SLP-84, No.6, 1-6, Dec. 2010. (Summary) 現在の音声対話システムは，一人のユーザに対して一つのエージェントが対応する 1 対 1 の対話を扱っているが，本論文ではシステム側のエージェントを複数にすることで多人数対話を実現するシステムを構築する．今回は，2 つのエージェントを扱った，三者対話システムの開発を行った．本システムは，我々がこれまでに構築した 1 対 1 対話の音声対話システムを拡張しており，ユーザに対する応答のタイミングや，応答の種類(あいづちなど)の制御を決定木を用いて行っている．また，ユーザからの非流暢な発話に対しても頑健に応答することが可能になっている．エージェントは， 2D のキャラクタを，それぞれ 2 台のディスプレイに 1 つずつ表示し，出力音声も別々のスピーカから出力されるようになっている．被験者実験の結果，被験者は三者対話による内容の幅の広がりや，対話の自然性を感じていることが示された．
106.	Ryota Nishimura and 中川聖一 : Spoken Dialog System using Multiple Dialog Agents, IPSJ SIG Technical Report, Vol.2010-SLP-80, No.7, 1-6, Feb. 2010. (Summary) 現在の音声対話システムは，一人のユーザに対して一つのエージェントが対応する 1 対 1 の対話を扱っているが，本論文ではシステム側のエージェントを複数にすることで多人数対話を実現するシステムを構築する．今回は，2 つのエージェントを扱った，三者対話システムの開発を行う．本システムは，我々がこれまでに構築した 1 対 1 対話の音声対話システムを拡張しており，ユーザに対する応答のタイミングや，応答の種類(あいづちなど)の制御を決定木を用いて行っている．また，ユーザからの非流暢な発話に対しても頑健に応答する事が可能になっている．エージェントは，2D のキャラクタを，それぞれ 2 台のディスプレイに 1 つずつ表示し，出力音声も別々のスピーカから出力されるようになっている．
107.	Ryota Nishimura and 中川聖一 : Spoken Dialog System Considering The Response Timing and The Evaluation, IPSJ SIG Technical Report, Vol.2009-SLP-77, No.22, 1-6, Jul. 2009. (Summary) 人間と機械が対話を行う際に，機械が人間同士の会話と同じように，自然な応答を返すことが出来れば，より円滑な対話を行うことが期待できる．本研究では，人間同士の雑談対話中にて生じる対話現象を模倣する音声対話システムを構築した．本システムでは，応答として，あいづち，復唱，共同補完などを扱っており，決定木を用いて応答種類と応答タイミングを決定している．また，本システムはユーザからのオーバーラップ応答(バージイン)や，非流暢な発話に対しても頑健に応答することが可能になっている．被験者実験の結果から，オーバーラップを含む通常応答やあいづちに対して高い自然性が示され，被験者の多くがあいづちに対して親しみを感じており，また，バージインは便利であると評価された．
108.	伊藤敏彦, Norihide Kitaoka and Ryota Nishimura : Influence of Response Timing in Speech Dialogues, IEICE Technical Report, Vol.108, No.283, 7-12, Nov. 2008. (Summary) 先行研究で行った対話リズムに関する分析結果の妥当性を確認するために，発話タイミング，分析結果に基づいた音声対話を作成し，対話の自然性や合成音の違和感発言の聞き取るなどを調べる知覚実験を行った．知覚実験1で合成音声による4段の短い目的指向対話を，知覚実験2では実音と合成音の1 分長の長めの雑談対話を用って実験を行った．その結果，発言では自然な発話が存在すること，発話内容に即した発話タイミングに自然性を感じるなど，対話リズムに関連する分析結果の妥当性を示すことができた．
109.	伊藤敏彦, Norihide Kitaoka and Ryota Nishimura : Subjective Experiments on Influence of Response Timing in Speech Dialogues, IPSJ SIG Technical Report, Vol.2008-SLP-72, No.18, 99-104, Jul. 2008. (Summary) 先行研究で行った対話リズムに関する分析結果の妥当性を確認するために，発話タイミング，分析結果に基づいた音声対話を作成し，対話の自然性や合成音の違和感発言の聞き取るなどを調べる知覚実験を行った．知覚実験1で合成音声による4段の短い目的指向対話を，知覚実験2では実音と合成音の1 分長の長めの雑談対話を用って実験を行った．その結果，発言では自然な発話が存在すること，発話内容に即した発話タイミングに自然性を感じるなど，対話リズムに関連する分析結果の妥当性を示すことができた．
110.	Ryota Nishimura, Norihide Kitaoka and 中川聖一 : Analysis of relationship between impression of human-human conversations and prosodic change and its modeling, 日本音響学会講演論文集, Vol.2-10-15, 1-4, Mar. 2008. (Summary) 人間と機械が対話を行う際に，機械が人間同士の会話と同じように，相手に同調を示すことができれば，より円滑な対話を行うことが期待できる．そのためには，実際の人間同士の対話の印象がどのような要因で決められるのかを把握し，その情報を用いて相手に良い印象を与えていくようにする必要がある．本研究では，韻律情報に着目し，人間同士の対話の印象と韻律変化との間にどのような関係性があるのかを分析した．そして，音声対話システムへの実装を目指した円滑に対話を行うための韻律制御モデルの構築を試みた．
111.	Ryota Nishimura, Norihide Kitaoka and 中川聖一 : Modeling prosodic change in conversations and its application for a spoken dialog system, 日本音響学会講演論文集, Vol.1-9-3, 1-2, Mar. 2007. (Summary) 人間と機械が対話を行う際に，機械が人間同士の会話と同じように，話者交代，割り込み，あいづちなどを自然に返すことが出来れば，より円滑な対話を行うことが期待できる．そのためには応答を返すタイミングや，出力音声の韻律情報を，実際の人間同士の対話のように制御する必要がある．本研究では，協調的な音声対話システムを実現するために，人間同士の対話における応答タイミングや韻律的な同調と，対話としての盛り上がり・意見の相違などとの関連を分析し，そのモデル化を試みた．また，そのモデルを音声対話システムに実装した．
112.	Ryota Nishimura, Norihide Kitaoka and 中川聖一 : Response Timing and Prosody Change Modeling in Conversations and Their Application to a Spoken Dialog System, 人工知能学会言語・音声理解と対話処理研究会, No.48, 37-42, Nov. 2006. (Summary) 本研究では，協調的な音声対話システムを実現するために，人間同士の対話における韻律的な同調と対話としての盛り上がりとの関連を分析し，そのモデル化を試みた．また，そのモデルを音声対話システムに搭載した．音声対話システムは，リアルタイムにあいづち，話者交替などの応答タイミングを検出し種々の雑談現象を扱い応答することが出来る雑談に向けた対話システムであり，タイミングの検出と応答の種類の決定には決定木を用い，応答を出力する際の韻律情報は，モデルを用いて制御している．
113.	Ryota Nishimura, Norihide Kitaoka and 中川聖一 : A spoken dialog system for chat-like conversations considering response timing, 人工知能学会言語・音声理解と対話処理研究会, No.46, 21-26, Mar. 2006. (Summary) 本研究では，リアルタイムにあいづち，話者交替などの応答タイミングを検出し種々の雑談現象を扱い応答することが出来る雑談に向けた対話システムの構築を行った．タイミングの検出と応答の種類の決定には決定木を用いており，その決定木の素性としては，言語情報と韻律情報を用いている．ポーズを検出せずに逐次的に処理をして応答を返すことから，オーバーラップした応答なども返すことが可能であり，実際の雑談に現れる様々な現象を実現できる．

Patent:

1.	中野有紀子, Ryota Nishimura and 眞鍋麟太郎 : DECENTRALIZED SYSTEM INTEGRATING AND SHARING INFORMATION ON THE BASIS OF ROS ARCHITECTURE, 2017-124154 (Jun. 2017), 2019-008158 (Jan. 2019), JP2019008158 (Jan. 2019).

Grants-in-Aid for Scientific Research (KAKEN Grants Database @ NII.ac.jp)

高齢者を対象とした永続的に利用できるマルチモーダル対話システム基盤技術の構築 (Project/Area Number: 23H00493 )
人間の感覚と整合する音声特徴空間の構築 (Project/Area Number: 22K19793 )
Spoken dialog system based on real-time control of dialogue tempo for smooth dialog (Project/Area Number: 19K04311 )
Development of base technologies for spoken dialog systems for elderlies (Project/Area Number: 19H01125 )
Search by Researcher Number (50635878)

Associate Professor : Nishimura, Ryota

Research

Personal Web Page

Field of Study

Subject of Study

Book / Paper

Book:

Academic Paper (Judged Full Paper):

Academic Letter:

Review, Commentary:

Proceeding of International Conference:

Proceeding of Domestic Conference:

Patent:

Grants-in-Aid for Scientific Research (KAKEN Grants Database @ NII.ac.jp)